On the Gross–Prasad conjecture with its refinement for (SO(5), SO(2)) and the generalized Böcherer conjecture

Abstract

We investigate the Gross–Prasad conjecture and its refinement for the Bessel periods in the case of $(\mathrm {SO}(5), \mathrm {SO}(2))$. In particular, by combining several theta correspondences, we prove the Ichino–Ikeda-type formula for any tempered irreducible cuspidal automorphic representation. As a corollary of our formula, we prove an explicit formula relating certain weighted averages of Fourier coefficients of holomorphic Siegel cusp forms of degree two, which are Hecke eigenforms, to central special values of $L$-functions. The formula is regarded as a natural generalization of the Böcherer conjecture to the non-trivial toroidal character case.

1. Introduction

The investigation of relations between periods of automorphic forms and special values of $L$-functions is a main focus of research in number theory. The central special values are of keen interest in light of the Birch and Swinnerton-Dyer conjecture and its generalizations.

Gross and Prasad [GP92, GP94] proclaimed a global conjecture relating non-vanishing of certain period integrals on special orthogonal groups to non-vanishing of central special values of certain tensor product $L$-functions, together with the local counterpart conjecture in the early 1990s. Later with Gan [GGP12], they extended the conjecture to classical groups and metaplectic groups. Meanwhile a refinement of the Gross–Prasad conjecture, which is a precise formula for the central special values of the tensor product $L$-functions for tempered cuspidal automorphic representations, was formulated by Ichino and Ikeda [II10] in the co-dimension-one special orthogonal case. Subsequently Harris [Har14] formulated a refinement of the Gan–Gross–Prasad conjecture in the co-dimension-one unitary case. Later an extension of the work of Ichino–Ikeda and Harris to the general Bessel period case was formulated by Liu [Liu16] and to the general Fourier–Jacobi period case for symplectic–metaplectic groups was formulated by Xue [Xue17].

In [FM17] we investigated the Gross–Prasad conjecture for Bessel periods for $\mathrm {SO}(2n+1)\times \mathrm {SO}(2)$ when the character on $\mathrm {SO}(2)$ is trivial, i.e. the special Bessel periods case and then, in the sequel [FM21], we proved its refinement, i.e. the Ichino–Ikeda-type precise $L$-value formula under the condition that the base field is totally real and all components at archimedean places are discrete series representations. As a corollary of our special value formula in [FM21], we obtained a proof of the long-standing conjecture by Böcherer in [Böc86], concerning central critical values of imaginary quadratic twists of spinor $L$-functions for holomorphic Siegel cusp forms of degree two which are Hecke eigenforms, thanks to the explicit calculations of the local integrals by Dickson, Pitale, Saha and Schmidt [DPSS20].

In this paper, for $(\mathrm {SO}(5), \mathrm {SO}(2))$, we vastly generalize the main results in [FM17, FM21]. Namely we prove the Gross–Prasad conjecture and its refinement for any Bessel period in the case of $(\mathrm {SO}(5), \mathrm {SO}(2))$. As a corollary, we prove the generalized Böcherer conjecture in the square-free case formulated in [DPSS20].

Let us introduce some notation and then state our main results precisely.

1.1 Notation

Let $F$ be a number field. We denote its ring of adeles by $\mathbb {A}_F$, which is mostly abbreviated as $\mathbb {A}$ for simplicity. Let $\psi$ be a non-trivial character of $\mathbb {A} / F$. For $a \in F^\times$, we denote by $\psi ^a$ the character of $\mathbb {A} / F$ defined by $\psi ^a(x) = \psi (ax)$. For a place $v$ of $F$, we denote by $F_v$ the completion of $F$ at $v$. When $v$ is non-archimedean, we denote by $\varpi _v$ and $q_v$ a uniformizer of $F_v$ and the cardinality of the residue field of $F_v$, respectively.

Let $E$ be a quadratic extension of $F$ and $\mathbb {A}_E$ be its ring of adeles. We denote by $x\mapsto x^\sigma$ the unique non-trivial automorphism of $E$ over $F$. Let us denote by $\mathrm {N}_{E / F}$ the norm map from $E$ to $F$. We choose $\eta \in E^\times$ such that $\eta ^\sigma =-\eta$ and fix. Let $d=\eta ^2$. We denote by $\chi _E$ the quadratic character of $\mathbb {A}^\times$ corresponding to the quadratic extension $E / F$. We fix a character $\Lambda$ of $\mathbb {A}_E^\times / E^\times$ whose restriction to $\mathbb {A}^\times$ is trivial once and for all.

1.2 Measures

Throughout the paper, for an algebraic group $\mathbf {G}$ defined over $F$, we let $\mathbf {G}_v$ denote $\mathbf {G}(F_v)$, the group of rational points of $\mathbf {G}$ over $F_v$, and we always take the measure $dg$ on $\mathbf {G}(\mathbb {A})$ to be the Tamagawa measure unless specified otherwise. For each $v$, we take the self-dual measure with respect to $\psi _v$ on $F_v$. Then recall that the product measure on $\mathbb {A}$ is the self-dual measure with respect to $\psi$ and is also the Tamagawa measure since $\mathrm {Vol}(\mathbb {A}/ F)=1$. For a unipotent algebraic group $\mathbf {U}$ defined over $F$, we also specify the local measure $du_v$ on $\mathbf {U}(F_v)$ to be the measure corresponding to the gauge form defined over $F$, together with our choice of the measure on $F_v$, at each place $v$ of $F$. Thus, in particular, we have

\[ du=\prod_v\,du_v\quad\text{and}\quad \mathrm{Vol}(\mathbf{U}(F)\backslash\mathbf{U}(\mathbb{A}), du)=1. \]

1.3 Similitudes

Various similitude groups appear in this article. Where there is no risk of confusion, we denote by $\lambda (g)$ the similitude of an element $g$ of a similitude group for simplicity.

1.4 Bessel periods

First we recall that when $V$ is a five-dimensional vector space over $F$ equipped with a non-degenerate symmetric bilinear form whose Witt index is at least one, there exists a quaternion algebra $D$ over $F$ such that

(1.4.1)\begin{equation} \mathrm{SO}(V)=\mathbb{G}_D, \end{equation}

where $\mathbb {G}_D=G_D/ Z_D$, $G_D$ is a similitude quaternionic unitary group over $F$ defined by

(1.4.2)\begin{equation} G_D(F): = \bigg\{g \in \mathrm{GL}_2(D) : {}^{t}\overline{g} \begin{pmatrix} 0 & 1\\ 1 & 0 \end{pmatrix} g= \lambda(g) \begin{pmatrix} 0 & 1\\ 1 & 0 \end{pmatrix},\ \lambda(g) \in F^\times \bigg\} \end{equation}

and $Z_D$ is the center of $G_D$. Here

\[ \overline{g}:=\begin{pmatrix} \overline{t} & \overline{u}\\ \overline{w} & \overline{v} \end{pmatrix}\quad\text{for}\ g=\begin{pmatrix} t & u\\ w & v \end{pmatrix}\in\mathrm{GL}_2(D), \]

where denoted by $x\mapsto \overline {x}$ for $x\in D$ is the canonical involution of $D$. In addition, we define a quaternionic unitary group $G_D^1$ over $F$ by

\[ G_D^1 := \{g \in G_D : \lambda(g) = 1\}.\label{G_D^1} \]

Let

\[ D^-:=\{x\in D: \mathrm{tr}_D(x)=0\}, \]

where $\mathrm {tr}_D$ denotes the reduced trace of $D$ over $F$. We recall that when $D\simeq \mathrm {Mat}_{2\times 2}(F)$, $G_D$ is isomorphic to the similitude symplectic group $\mathrm {GSp}_2$ which we denote by $G$, i.e.

(1.4.3)\begin{equation} G(F):=\bigg\{g\in\mathrm{GL}_4(F):{}^tg\begin{pmatrix} 0 & 1_2\\ -1_2 & 0 \end{pmatrix}g=\lambda(g)\begin{pmatrix} 0 & 1_2\\ -1_2 & 0\end{pmatrix},\ \lambda(g)\in F^\times\bigg\}. \end{equation}

In addition, we define the symplectic group $\mathrm {Sp}_2$, which we denote by $G^1$, as

\[ G^1 := \{g \in G : \lambda(g) =1\}.\label{G^1} \]

We let $\mathrm {PGSp}_2 = G / Z_G$ by $\mathbb {G}$, where $Z_G$ denotes the center of $G$. Thus, when $D$ is split, $G_D\simeq G=\mathrm {GSp}_2$, $G_D^1 \simeq G^1 = \mathrm {Sp}_2$ and $\mathbb {G}_D\simeq \mathbb {G}=\mathrm {PGSp}_2$.

The Siegel parabolic subgroup $P_D$ of $G_D$ has the Levi decomposition $P_D = M_{D} N_{D}$ where

\[ M_{D}(F): = \bigg\{\!\begin{pmatrix} x & 0\\ 0 & \mu \cdot x\end{pmatrix} : x \in D^\times,\ \mu \in F^\times\bigg\},\quad N_{D}(F): = \bigg\{\!\begin{pmatrix} 1 & u\\ 0 & 1 \end{pmatrix} : u \in D^-\bigg\}.\label{d:N_D} \]

For $\xi \in D^-(F)$, let us define a character $\psi _\xi$ on $N_D(\mathbb {A})$ by

(1.4.4)\begin{equation} \psi_\xi\begin{pmatrix} 1 & u\\ 0 & 1 \end{pmatrix}:=\psi(\mathrm{tr}_D(\xi u)). \end{equation}

We note that for $\big (\begin{smallmatrix}x & 0\\0 & \mu \cdot x\end{smallmatrix}\big )\in M_D(F)$, we have

(1.4.5)\begin{equation} \psi_\xi\bigg[\begin{pmatrix} x & 0\\ 0 & \mu\cdot x \end{pmatrix} \begin{pmatrix} 1 & u\\ 0 & 1 \end{pmatrix}\begin{pmatrix} x & 0\\ 0 & \mu\cdot x \end{pmatrix}^{-1}\bigg]=\psi_{\mu^{-1}\cdot x^{-1}\xi x}\begin{pmatrix}1 & u\\0 & 1\end{pmatrix}. \end{equation}

Suppose that $F(\xi )\simeq E$. Let us define a subgroup $T_\xi$ of $D^\times$ by

(1.4.6)\begin{equation} T_\xi:=\{x\in D^\times: x\,\xi \,x^{-1}=\xi\}. \end{equation}

Then since $F(\xi )$ is a maximal commutative subfield of $D$, we have

(1.4.7)\begin{equation} T_\xi(F)=F(\xi)^\times\simeq E^\times. \end{equation}

We identify $T_\xi$ with the subgroup of $M_D$ given by

(1.4.8)\begin{equation} \bigg\{\!\begin{pmatrix}x & 0\\0 & x\end{pmatrix}: x\in T_\xi\bigg\}. \end{equation}

We note that by (1.4.5), we have

\[ \psi_\xi(tnt^{-1})=\psi_\xi(n)\quad \text{for $t\in T_\xi(\mathbb{A})$ and $n\in N_D(\mathbb{A})$}. \]

We define the Bessel subgroup $R_{\xi }$ of $G_D$ by

(1.4.9)\begin{equation} R_\xi:=T_\xi N_D. \end{equation}

Then the Bessel periods defined below are indeed the periods in question in the Gross–Prasad conjecture for $(\mathrm {SO}(5), \mathrm {SO}(2))$.

Definition 1.1 Let $\pi$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ whose central character is trivial and $V_\pi$ its space of automorphic forms. Let $\Lambda$ be a character of $\mathbb {A}_E^\times / E^\times$ whose restriction to $\mathbb {A}^\times$ is trivial. Let $\xi \in D^-(F)$ such that $F(\xi )\simeq E$. Fix an $F$-isomorphism $T_\xi \simeq E^\times$ and regard $\Lambda$ as a character of $T_\xi (\mathbb {A}) / T_\xi (F)$. We define a character $\chi ^{\xi,\Lambda }$ on $R_\xi (\mathbb {A})$ by

(1.4.10)\begin{equation} \chi^{\xi,\Lambda}(tn):=\Lambda(t)\psi_\xi(n)\quad \text{for $t\in T_\xi(\mathbb{A})$ and $n\in N_D(\mathbb{A})$}. \end{equation}

Then for $f\in V_\pi$, we define $B_{\xi, \Lambda,\psi }(f)$, the $(\xi, \Lambda,\psi )$-Bessel period of $f$, by

(1.4.11)\begin{equation} B_{\xi, \Lambda,\psi}(f):=\int_{\mathbb{A}^\times R_\xi(F) \backslash R_\xi(\mathbb{A})} f(r)\chi^{\xi,\Lambda}(r)^{-1}\, dr. \end{equation}

We say that $\pi$ has the $(\xi, \Lambda, \psi )$-Bessel period when the linear form $B_{\xi,\Lambda, \psi }$ is not identically zero on $V_\pi$.

Remark 1.1 Here we record the dependency of $B_{\xi,\Lambda,\psi }$ on the choices of $\xi$ and $\psi$. First we note that for $\xi ^\prime \in D^-(F)$, we have $F(\xi ^\prime )\simeq E$ if and only if

(1.4.12)\begin{equation} \xi^\prime=\mu\cdot \alpha^{-1} \xi\alpha \quad\text{for some $\alpha\in D^\times(F)$ and $\mu\in F^\times$} \end{equation}

by the Skolem–Noether theorem. Suppose that $\xi ^\prime \in D^-(F)$ satisfies (1.4.12) and $\psi ^\prime =\psi ^a$ where $a\in F^\times$. Let $m_0=\big (\begin{smallmatrix}\alpha & 0\\ 0 & a^{-1}\mu \cdot \alpha \end{smallmatrix}\big ) \in M_D(F)$. Then by (1.4.5), we have

(1.4.13)\begin{align} B_{\xi, \Lambda, \psi}(\pi(m_0)f) &= \int_{\mathbb{A}^\times T_{\xi^\prime}(F) \backslash T_{\xi^\prime}(\mathbb{A})}\int_{N_D(F)\backslash N_D(\mathbb{A})} f(t^\prime n^\prime)\Lambda(t^{\prime})^{-1} \psi^\prime_{\xi^\prime}(n^\prime)\, dt^\prime\, dn^\prime \nonumber\\ &= B_{\xi^\prime,\Lambda, \psi^\prime}(f) , \end{align}

where we identify $T_{\xi ^\prime }(F)$ with $E^\times$ via the $F$-isomorphism $F(\xi ^\prime )\ni x\mapsto \alpha x\alpha ^{-1}\in F(\xi ) \simeq E$.

Definition 1.2 Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ whose central character is trivial. Let $\Lambda$ be a character of $\mathbb {A}_E^\times / E^\times$ whose restriction to $\mathbb {A}^\times$ is trivial. Then we say that $\pi$ has the $(E,\Lambda )$-Bessel period if there exist $\xi \in D^-(F)$ such that $F(\xi )\simeq E$ and a non-trivial character $\psi$ of $\mathbb {A}/ F$ so that $\pi$ has the $(\xi,\Lambda,\psi )$-Bessel period. This terminology is well-defined because of the relation (1.4.13).

1.5 Gross–Prasad conjecture

First we introduce the following definition which is inspired by the notion of local $G$-equivalence in Hiraga and Saito [HS12, p. 23].

Definition 1.3 Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ whose central character is trivial. Let $D^\prime$ be a quaternion algebra over $F$ and $(\pi ^\prime, V_{\pi ^\prime })$ an irreducible cuspidal automorphic representation of $G_{D^\prime }(\mathbb {A})$. Then we say that $\pi$ is locally $G^+$-equivalent to $\pi ^\prime$ if at almost all places $v$ of $F$ where $D(F_v)\simeq D^\prime (F_v)$, there exists a character $\chi _v$ of $G_D(F_v) / G_D(F_v)^+$ such that $\pi _v \otimes \chi _v \simeq \pi _v^\prime$. Here

(1.5.1)\begin{equation} G_D(F)^+:=\{g\in G_D(F): \lambda(g)\in \mathrm{N}_{E/ F}(E^\times)\}. \end{equation}

Remark 1.2 When $\pi$ and $\pi ^\prime$ have weak functorial lifts to $\mathrm {GL}_4(\mathbb {A})$, say $\Pi$ and $\Pi ^\prime$, respectively, the notion of local $G^+$-equivalence is described simply as the following. Suppose that $\pi$ and $\pi ^\prime$ are locally $G^+$-equivalent. Then there exists a character $\omega$ of $G_D(\mathbb {A})$ such that $\pi \otimes \omega$ is nearly equivalent to $\pi ^\prime$, where $\omega$ may not be automorphic. Since $\omega _v$ is either $\chi _{E_v}$ or trivial at almost all places $v$ of $F$, we have $\mathrm {BC}_{E/ F}(\Pi )\simeq \mathrm {BC}_{E/ F}(\Pi ^\prime )$ where $\mathrm {BC}_{E/ F}$ denotes the base change lift to $\mathrm {GL}_4(\mathbb {A}_E)$. Then by a result of Arthur and Clozel [AC89, Theorem 3.1], we have $\Pi \simeq \Pi ^\prime$ or $\Pi ^\prime \otimes \chi _E$. Hence, $\pi$ is nearly equivalent to either $\pi ^\prime$ or $\pi ^\prime \otimes \chi _E$. The converse is clear.

Then our first main result is on the Gross–Prasad conjecture for $(\mathrm {SO}(5), \mathrm {SO}(2))$.

Theorem 1.1 Let $E$ be a quadratic extension of $F$. Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ with a trivial central character and $\Lambda$ a character of $\mathbb {A}_E^\times / E^\times$ whose restriction to $\mathbb {A}^\times$ is trivial.

1. (i) Suppose that $\pi$ has the $(E,\Lambda )$-Bessel period. Moreover, assume that:

   (1.5.2)\begin{align} & \text{there exists a finite place $w$ of $F$ such that} \nonumber\\ &\qquad\qquad\qquad\qquad \text{$\pi_w$ and its local theta lift to $\mathrm{GSO}_{4,2}(F_w)$ are generic.} \end{align}
   Here $\mathrm {GSO}_{4,2}$ denotes the identity component of $\mathrm {GO}_{4,2}$, the similitude orthogonal group associated to the six-dimensional orthogonal space $(E, \mathrm {N}_{E / F}) \oplus \mathbb {H}^2$ over $F$ where $\mathbb {H}$ denotes the hyperbolic plane over $F$.

   Then there exists a finite set $S_0$ of places of $F$ containing all archimedean places of $F$ such that the partial $L$-function

   (1.5.3)\begin{equation} L^S \big(\tfrac{1}{2}, \pi \times \mathcal{AI} (\Lambda)\big) \ne 0 \end{equation}
   for any finite set $S$ of places of $F$ with $S\supset S_0$. Here, $\mathcal {AI}(\Lambda )$ denotes the automorphic induction of $\Lambda$ from $\mathrm {GL}_1(\mathbb {A}_E)$ to $\mathrm {GL}_2(\mathbb {A})$. Moreover, there exists a globally generic irreducible cuspidal automorphic representation $\pi ^\circ$ of $G(\mathbb {A})$ which is locally $G^+$-equivalent to $\pi$.

2. (ii) Assume that:

   (1.5.4)\begin{align} & \text{the endoscopic classification of Arthur,} \nonumber\\ &\qquad\qquad\quad \text{i.e. [Art13, Conjectures~9.4.2, 9.5.4] holds for $\mathbb{G}_{D_\circ}$.} \end{align}
   Here $D_\circ$ denotes an arbitrary quaternion algebra over $F$.

   Suppose that $\pi$ has a generic Arthur parameter, namely the parameter is of the form $\Pi _0$ or $\Pi _1 \boxplus \Pi _2$ where $\Pi _i$ is an irreducible cuspidal automorphic representation of $\mathrm {GL}_4(\mathbb {A})$ for $i=0$ and of $\mathrm {GL}_2(\mathbb {A})$ for $i=1,2$, respectively, such that $L(s, \Pi _i, \wedge ^2)$ has a pole at $s=1$.

   Then we have

   (1.5.5)\begin{equation} L \big(\tfrac{1}{2}, \pi \times \mathcal{AI} (\Lambda)\big) \ne 0 \end{equation}
   if and only if there exists a pair $(D^\prime, \pi ^\prime )$ where $D^\prime$ is a quaternion algebra over $F$ containing $E$ and $\pi ^\prime$ an irreducible cuspidal automorphic representation of $G_{D^\prime }$ which is nearly equivalent to $\pi$ such that $\pi ^\prime$ has the $(E,\Lambda )$-Bessel period.

   Moreover, when $\pi$ is tempered, the pair $(D^\prime, \pi ^\prime )$ is uniquely determined.

Remark 1.3 In (1.5.5), $L (s, \pi \times \mathcal {AI} (\Lambda ))$ denotes the complete $L$-function defined as the following.

When $\mathcal {AI} (\Lambda )$ is not cuspidal, i.e. $\Lambda =\Lambda _0 \circ \mathrm {N}_{E / F}$ for a character $\Lambda _0$ of $\mathbb {A}^\times / F^\times$, we define

\[ L(s,\pi \times \mathcal{AI} (\Lambda)) : = L(s,\pi \times \Lambda_0) L(s,\pi \times \Lambda_0 \chi_E), \]

where each factor on the right-hand side is defined by the doubling method as in Lapid and Rallis [LR05] or Yamana [Yam14].

When $\mathcal {AI} (\Lambda )$ is cuspidal, the partial $L$-function $L^S(s,\pi \times \mathcal {AI}(\Lambda ))$ may be defined by Theorem C.1 in Appendix C for a finite set $S$ of places of $F$ such that $\pi _v$ and $\Pi (\Lambda )_v$ are unramified at $v \not \in S$. Further, we define the local $L$-factor at each place $v \in S$ by the local Langlands parameters for $\pi _v$ and $\Pi (\Lambda )_v$, where the local Langlands parameters are given by Gan and Takeda [GT11a] for $\mathbb {G}(F_v)$ (also Arthur [Art13]), Gan and Tantono [GT14] for $\mathbb {G}_D(F_v)$, Kutzko [Kut80] for $\mathrm {GL}_2(F_v)$ at finite places and by Langlands [Lan89] at archimedean places.

We note that the condition (1.5.3) and the condition (1.5.5) are equivalent from the definition of local $L$-factors when $\pi$ is tempered.

Remark 1.4 Suppose that at a finite place $w$ of $F$, the group $G_D(F_w)$ is split and the representation $\pi _w$ is generic and tempered. Then by Gan and Ichino [GI14, Proposition C.4], the big theta lift of $\pi _w$ and the local theta lift of $\pi _w$ coincide. Thus, the genericity of the local theta lift of $\pi _w$ follows from Gan and Takeda [GT11b, Corollary 4.4] for the dual pair $(G, \mathrm {GSO}_{3,3})$ and from a local analogue of the computations in [Mor14a, § 3.1] for the dual pair $(G^+,\mathrm {GSO}_{4,2})$, respectively. Here

(1.5.6)\begin{equation} G(F)^+:=\{g\in G: \lambda(g)\in\mathrm{N}_{E/ F}(E^\times)\}. \end{equation}

When a local representation $\pi _w$ is unramified and tempered, $\pi _w$ is generic as remarked in [FM17, Remark 2]. Hence, the assumption (1.5.2) is fulfilled when $\pi$ is tempered.

In our previous paper [FM17], Theorem 1.1 for the pair $(\mathrm {SO}(2n+1),\mathrm {SO}(2))$ was proved when $\Lambda$ is trivial. Meanwhile Jiang and Zhang [JZ20] studied the Gross–Prasad conjecture in a very general setting assuming the endoscopic classification of Arthur, in general, by using the twisted automorphic descent. Though Theorem 1.1 is subsumed in [JZ20] as a special case, we believe that our method, which is different from theirs, has its own merits because of its concreteness. We also note that because of the temperedness of $\pi$, the uniqueness of the pair $(D^\prime,\pi ^\prime )$ in Theorem 1.1(ii) follows from the local Gan–Gross–Prasad conjecture for $(\mathrm {SO}(5), \mathrm {SO}(2))$ by Prasad and Takloo-Bighash [PT11, Theorem 2] (see also Waldspurger [Wal85] in the general case) at finite places and by Luo [Luo20] at archimedean places. We shall give another proof of this uniqueness by reducing it to a similar assertion in the unitary group case.

1.6 Refined Gross–Prasad conjecture

Let $(\pi, V_\pi )$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with trivial central character. For $\phi _1, \phi _2 \in V_{\pi }$, we define the Petersson inner product $(\phi _1,\phi _2)_{\pi }$ on $V_\pi$ by

\[ (\phi_1, \phi_2)_{\pi} = \int_{Z_D(\mathbb{A}) G_D(F) \backslash G_D(\mathbb{A})} \phi_1(g) \overline{\phi_2(g)} \,dg, \]

where $dg$ denotes the Tamagawa measure. Then at each place $v$ of $F$, we take a $G_D(F_v)$-invariant hermitian inner product on $V_{\pi _v}$ so that we have a decomposition $(\,{,}\,)_\pi =\prod _v(\,{,}\,)_{\pi _v}$. In the definition of the Bessel period (1.4.11), we take $dr=dt\,du$ where $dt$ and $du$ are the Tamagawa measures on $T_\xi (\mathbb {A})$ and $N_D(\mathbb {Z})$, respectively. We take and fix the local measures $du_v$ and $dt_v$ so that $du=\prod _v du_v$ and

(1.6.1)\begin{equation} dt = C_\xi \prod dt_v \end{equation}

where $C_\xi$ is a constant called the Haar measure constant in [II10]. Then the local Bessel period $\alpha ^{\xi,\Lambda }_v:V_{\pi _v}\times V_{\pi _v}\to \mathbb {C}$ and the local hermitian inner product $(\,{,}\,)_{\pi _v}$ are defined as in § 2.4.

Suppose that $D$ is not split. Then by Li [Li92], there exists a pair $(\xi ^\prime, \Lambda ^\prime )$ such that $\pi$ has the $(\xi ^\prime,\Lambda ^\prime, \psi )$-Bessel period. Here $\xi ^\prime \in D^-(F)$ such that $E^\prime :=F(\xi ^\prime )$ is a quadratic extension of $F$ and $\Lambda ^\prime$ is a character on $\mathbb {A}_{E^\prime }^\times / \mathbb {A}^\times {E^\prime }^\times$. Then by Proposition 4.1, which is a consequence of the proof of Theorem 1.1(i), there exists an irreducible cuspidal automorphic representation $\pi ^\circ$ of $G(\mathbb {A})$ which is generic and locally $G^+$-equivalent to $\pi$. We take the functorial lift of $\pi ^\circ$ to $\mathrm {GL}_4 (\mathbb {A})$ by Cogdell, Kim, Piatetski-Shapiro and Shahidi [CKPS04], which is of the form $\Pi _1 \boxplus \cdots \boxplus \Pi _{\ell _0}$ with $\Pi _i$ an irreducible cuspidal automorphic representation of $\mathrm {GL}_{m_i}(\mathbb {A})$ for each $i$. Then we define an integer $\ell (\pi )$ by $\ell (\pi )=\ell _0$. We note that $\pi ^\circ$ may not be unique, but $\ell (\pi )$ does not depend on the choice of the pair $(\xi ^\prime,\Lambda ^\prime )$ by Proposition 4.1 and Lemmas 4.2 and 4.3, and thus it depends only on $(\pi, V_\pi )$. When $D$ is split, then $\pi$ has the functorial lift to $\mathrm {GL}_4(\mathbb {A})$ by Arthur [Art13] (see also Cai, Friedberg and Kaplan [CFK18]) and we define $\ell (\pi )$ in a similar way.

Our second main result is the refined Gross–Prasad conjecture formulated by Liu [Liu16], i.e. the Ichino–Ikeda-type explicit central value formula, in the case of $(\mathrm {SO}(5), \mathrm {SO}(2))$.

Theorem 1.2 Let $(\pi, V_\pi )$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with a trivial central character.

Then for any non-zero decomposable cusp form $\phi =\otimes _v\phi _v\in V_\pi$, we have

(1.6.2)\begin{equation} \frac{|B_{\xi, \Lambda,\psi}(\phi)|^2}{( \phi,\phi )_\pi} =2^{-\ell(\pi)}\,C_{\xi}\cdot \bigg(\prod_{j=1}^2\zeta_F(2j)\bigg) \frac{L\big(\frac{1}{2},\pi \times \mathcal{AI} (\Lambda)\big)}{L(1,\pi,\mathrm{Ad})L(1,\chi_E)} \cdot\prod_v\frac{\alpha_v^\natural(\phi_v)}{ (\phi_v,\phi_v)_{\pi_v}}. \end{equation}

Here $\zeta _F(s)$ denotes the complete zeta function of $F$ and $\alpha _v^\natural (\phi _v)$ is defined by

\[ \alpha_v^\natural(\phi_v) =\frac{L(1,\pi_v,\mathrm{Ad}) L(1,\chi_{E,v})}{L(1/2,\pi_v \times \Pi (\Lambda)_v) \prod_{j=1}^2\zeta_{F_v}(2j)}\cdot \alpha_{\Lambda_v, \psi_{\xi, v}}(\phi_v,\phi_v). \]

We note that ${\alpha _v^\natural (\phi _v)}/{(\phi _v,\phi _v)_{\pi _v}}=1$ for almost all places $v$ of $F$ by [Liu16].

Remark 1.5 Under the assumption (1.5.4), we have $|\mathcal {S}(\phi _\pi )| =2^{\ell (\pi )}$, where $\phi _\pi$ denotes the Arthur parameter of $\pi$ and $\mathcal {S}(\phi _\pi )$ the centralizer of $\phi _\pi$ in the complex dual group $\hat {G}$. Hence, (1.6.2) coincides with the conjectural formula in Liu [Liu16, Conjecture 2.5 (3)]. Thus, when $D$ is split, i.e. $G_D\simeq G$, our theorem proves Liu's conjecture since the assumption (1.5.4) is indeed fulfilled. After submitting this paper, Ishimoto posted a preprint [Ish24] on arXiv, in which he gives the endoscopic classification of representations of non-quasi-split orthogonal groups for generic Arthur parameters. Hence, our theorem proves [Liu16, Conjecture 2.5 (3)] completely in the case of $(\mathrm {SO}(5), \mathrm {SO}(2))$.

Remark 1.6 Let $\pi _{\rm gen}$ denote the irreducible cuspidal globally generic automorphic representation of $G(\mathbb {A})$ which has the same $L$-parameter as $\pi$. When $\pi _v$ is unramified at any finite place $v$ of $F$, Chen and Ichino [CI23] proved an explicit formula of the ratio $L(1,\pi,\mathrm {Ad})/ (\Phi _{\rm gen}, \Phi _{\rm gen})$ for a suitably normalized cusp form $\Phi _{\rm gen}$ in the space of $\pi _{\rm gen}$.

Remark 1.7 In the unitary case, remarkable progress has been made in the Gan–Gross–Prasad conjecture and its refinement for Bessel periods, by studying the Jacquet–Rallis relative trace formula. In the striking paper [BLZZ21] by Beuzart-Plessis, Liu, Zhang and Zhu, a proof in the co-dimension-one case for irreducible cuspidal tempered automorphic representations of unitary groups such that their base change lifts are cuspidal was given by establishing an ingenious method to isolate the cuspidal spectrum. In yet another striking paper by Beuzart-Plessis, Chaudouard and Zydor [BCZ22], a proof for all endoscopic cases in the co-dimension one setting was given by a precise study of the relative trace formula. Very recently, in a remarkable preprint by Beuzart-Plessis and Chaudouard [BC23], the above results are extended to arbitrary co-dimension cases. Thus the Gan–Gross–Prasad conjecture and its refinement for Bessel periods on unitary groups are now proved in general.

In contrast, the orthogonal case in general is still open. We note that, in the $(\mathrm {SO}(5),\mathrm {SO}(2))$ case, the first author has formulated relative trace formulas to approach the formula (1.6.2) and proved the fundamental lemmas in his joint work with Shalika [FS03], Martin [FM11] and Martin and Shalika [FM13a]. In order to deduce the $L$-value formula from these relative trace formulas, several issues such as smooth transfer of test functions must be overcome. In the above-mentioned co-dimension-one unitary group case, reductions to Lie algebras played crucial roles to solve similar issues. However Bessel periods in our case involves integration over unipotent subgroups and it is not clear, at least to the first author, how to make the reduction to Lie algebras work.

Remark 1.8 In the co-dimension-one orthogonal group case, the refined Gross–Prasad conjecture has been deduced from the Waldspurger formula [Wal85] in the $(\mathrm {SO}(3), \mathrm {SO}(2))$ case and from the Ichino formula [Ich08] in the $(\mathrm {SO}(4), \mathrm {SO}(3))$ case, respectively. Gan and Ichino [GI11] studied the $(\mathrm {SO}(5), \mathrm {SO}(4))$-case when the representation of $\mathrm {SO}(5)$ is a theta lift from $\mathrm {GSO}(4)$ by reduction to the $(\mathrm {SO}(4), \mathrm {SO}(3))$ case.

Liu [Liu16] proved Theorem 1.2 when $D$ is split and $\pi$ is an endoscopic lift, i.e. a Yoshida lift, by reducing it to the Waldspurger formula [Wal85]. The case when $\pi$ is a non-endoscopic Yoshida lift was proved later by Corbett [Cor17] in a similar manner.

As a corollary of Theorem 1.2, we prove the $(\mathrm {SO}(5), \mathrm {SO}(2))$ case of the Gan–Gross–Prasad conjecture in the form as stated in [GGP12, Conjecture 24.1].

Corollary 1.1 Let $(\pi, V_\pi )$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with a trivial central character. Then the following three conditions are equivalent:

1. (i) the $(\xi,\Lambda, \psi )$-Bessel period does not vanish on $\pi$;

2. (ii) $L (\frac {1}{2}, \pi \times \mathcal {AI}(\Lambda )) \ne 0$ and the local Bessel period $\alpha _{\Lambda _v, \psi _{\xi, v}}\not \equiv 0$ on $\pi _v$ at any place $v$ of $F$;

3. (iii) $L (\frac {1}{2}, \pi \times \mathcal {AI} (\Lambda )) \ne 0$ and $\mathrm {Hom}_{R_{\xi,v}}(\pi _v,\chi ^{\xi,\Lambda }_v) \ne \{0\}$ at any place $v$ of $F$.

Remark 1.9 The equivalence between conditions (i) and (ii) is immediate from Theorem 1.2. The equivalence

(1.6.3)\begin{equation} \alpha_{\Lambda_v, \psi_{\xi, v}}\not\equiv 0\Longleftrightarrow \mathrm{Hom}_{R_{\xi,v}}(\pi_v,\chi^{\xi,\Lambda}_v)\ne\{0\} \end{equation}

is proved by Waldspurger [Wal12b] at any non-archimedean place $v$ and by Luo [Luo20] recently at any archimedean place $v$, respectively.

1.7 Method

In [FM17, FM21] we used the theta correspondence for the dual pair $(\mathrm {SO}(2n+1), \mathrm {Mp}_n)$.

The main tool in [FM17] was the pull-back formula by the first author [Fur95] for the Whittaker period on $\mathrm {Mp}_n$, which is expressed by a certain integral involving the Special Bessel period on $\mathrm {SO}(2n+1)$. This forced us the restriction that the character $\Lambda$ on $\mathrm {SO}(2)$ is trivial.

In [FM21], to prove the refined Gross–Prasad conjecture for $(\mathrm {SO}(2n+1),\mathrm {SO}(2))$ when $\Lambda$ is trivial, the following additional restrictions were necessary.

1. (i) The base field $F$ is totally real and at every archimedean place $v$ of $F$, the representation $\pi _v$ is a discrete series representation.

2. (ii) The assumption (1.5.4).

An additional main tool needed in [FM21] was the Ichino–Ikeda-type formula for the Whittaker periods on $\mathrm {Mp}_n$ by Lapid and Mao [LM17], which imposed on us condition (i). In fact, their proof was to reduce the global identity to certain local identities. They proved the local identities in general at non-archimedean places. On the other hand, at archimedean places, their proof was to note the equivalence between their local identities and the formal degree conjecture by Hiraga, Ichino and Ikeda [HII08a, HII08b] and then to prove the latter when $\pi$ is a discrete series representation. Our proof in [FM21] was to reduce to the case when $\pi$ has the special Bessel period by the assumption (1.5.4) and to combine these two main tools with the Siegel–Weil formula.

It does not seem plausible that a straightforward generalization of the method of [FM17, FM21] would allow us to remove these restrictions. Thus, we need to adopt a new strategy in this paper.

Our main method here is again theta correspondence but we use it differently and in a more intricate way. First we consider the quaternionic dual pair $(G^+_D,\mathrm {GSU}_{3,D})$ where $\mathrm {GSU}_{3,D}$ denotes the identity component of the similitude quaternion unitary group $\mathrm {GU}_{3,D}$ defined by (2.1.9) and $G^+_D$ defined by (1.5.1). Then we recall the accidental isomorphism

(1.7.1)\begin{equation} \mathrm{PGSU}_{3,D}\simeq \mathrm{PGU}_{4,\varepsilon} \end{equation}

when $D\simeq D_\varepsilon$ given by (2.1.1) and $\mathrm {GU}_{4,\varepsilon }$ is the similitude unitary group defined by (2.1.14). Hence, we have

(1.7.2)\begin{equation} \mathrm{GU}_{4,\varepsilon}\simeq \begin{cases} \mathrm{GU}_{2,2}, & \text{when $D$ is split, i.e. $\varepsilon\in\mathrm{N}_{E/ F} (E^\times)$};\\ \mathrm{GU}_{3,1}, & \text{when $D$ is non-split, i.e. $\varepsilon\notin\mathrm{N}_{E/ F} (E^\times)$}. \end{cases} \end{equation}

Thus, our theta correspondence for $(G^+_D,\mathrm {GSU}_{3,D})$ induces a correspondence for the pair $(\mathbb {G}_D,\mathrm {PGU}_{4,\varepsilon })$. Then we note that the pull-back of a certain Bessel period on $\mathrm {PGU}_{4,\varepsilon }$ is an integral involving the $(\xi,\Lambda, \psi )$-Bessel period on $G_D$.

Theorem 1.1 is reduced essentially to the Gan–Gross–Prasad conjecture for the Bessel periods on $\mathrm {GU}_{4,\varepsilon }$, which we proved in [FM22] using the theta correspondence for the pair $(\mathrm {GU}_{4,\varepsilon },\mathrm {GU}_{2,2})$.

Similarly Theorem 1.2 is reduced to the refined Gan–Gross–Prasad conjecture for the Bessel periods on $\mathrm {GU}_{4,\varepsilon }$. For the reader's sake, here we present an outline of the proof when the $(\xi,\Lambda,\psi )$-Bessel period does not vanish. Note that in the following paragraph the notation used is provisional and the argument is not rigorous since our intention here is to present a rough sketch of the main idea.

Let $(\pi, V_\pi )$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with a trivial central character. Suppose that the $(\xi,\Lambda,\psi )$-Bessel period, which we denote by $B$, does not vanish on $\pi$. Let $\theta (\pi )$ be the theta lift of $\pi$ to $\mathrm {GSU}_{3,D}$. When $G_D=G$ and the theta lift of $\pi$ to $\mathrm {GSO}_{3,1}$ is non-zero, $\theta (\pi )$ is not cuspidal but the explicit formula (1.6.2) has been already proved by Corbett [Cor17]. Thus, suppose otherwise. Then $\theta (\pi )$ is a non-zero irreducible cuspidal tempered automorphic representation. The pull-back of a certain Bessel period, which we denote by $\mathcal {B}$ on $\mathrm {GSU}_{3,D}$ is written as an integral involving $B$. As in our previous paper [FM21], the explicit formula for $B$ is reduced to that for $\mathcal {B}$, which we obtain in the following steps.

1. (1) Via the isomorphism (1.7.1), regard $\theta (\pi )$ as an automorphic representation of $\mathrm {GU}_{4,\varepsilon }$ and then consider its theta lift $\theta _\Lambda (\theta (\pi ))$, which depends on $\Lambda$, to $\mathrm {GU}_{2,2}$. The temperedness of $\pi$ implies that $\theta _\Lambda (\theta (\pi ))$ is an irreducible cuspidal automorphic representation of $\mathrm {GU}_{2,2}$. Then the pull-back of a certain Whittaker period $\mathcal {W}$ on $\mathrm {GU}_{2,2}$ is written as an integral involving the Bessel period $\mathcal {B}$. Then in [FM22], it is shown that the explicit formula for $\mathcal {B}$ follows from that for $\mathcal {W}$. Thus, we are reduced to show the explicit formula for $\mathcal {W}$.

2. (2) Via the isomorphism $\mathrm {PGU}_{2,2}\simeq \mathrm {PGSO}_{4,2}$, regard $\theta _\Lambda (\theta (\pi ))$ as an automorphic representation of $\mathrm {GSO}_{4,2}$. Let $\pi ^\prime$ be the theta lift of $\theta _\Lambda (\theta (\pi ))$ to $G=\mathrm {GSp}_2$. Then it is shown that $\pi ^\prime$ is a globally generic cuspidal automorphic representation of $G$ and indeed the pull-back of the Whittaker period $W$ on $G$ is expressed as an integral involving $\mathcal {W}$. Hence, we are reduced to the explicit formula for $W$.

3. (3) Since the theta lift of the globally generic cuspidal automorphic representation $\pi ^\prime$ of $G$ to either $\mathrm {GSO}_{2,2}$ or $\mathrm {GSO}_{3,3}$ is non-zero and cuspidal, we are further reduced to the explicit formulas for the Whittaker periods on $\mathrm {PGSO}_{2,2}$ and $\mathrm {PGSO}_{3,3}$ by the pull-back computation.

4. (4) Recall the accidental isomorphisms $\mathrm {PGSO}_{2,2}\simeq \mathrm {PGL}_2\times \mathrm {PGL}_2$, $\mathrm {PGSO}_{3,3}\simeq \mathrm {PGL}_4$. Since the explicit formula for the Whittaker period on $\mathrm {PGL}_n$ is already proved by Lapid and Mao [LM15], we are done.

Remark 1.10 Though we only consider the case when $\mathrm {SO}(2)$ is non-split in this paper, the split case is proved by a similar argument as follows. First we note that $D$ is necessarily split when $\mathrm {SO}(2)$ is split and, hence, $G_D\simeq G$. If the theta lift to $\mathrm {GSO}_{2,2}$ is non-zero, it is a Yoshida lift and Liu [Liu16] proved the explicit formula. Suppose otherwise. Then the theta lift to $\mathrm {GSO}_{3,3}$ is non-zero and cuspidal. The pull-back of a certain Bessel period on $\mathrm {GSO}_{3,3}$ is an integral involving the split Bessel period on $G$ (see § 3.1.2). We recall the accidental isomorphism $\mathrm {PGSO}_{3,3} \simeq \mathrm {PGL}_4$. We consider the theta correspondence for the pair $(\mathrm {GL}_4,\mathrm {GL}_4)$ instead of $(\mathrm {GU}_{4,\varepsilon },\mathrm {GU}_{4,\varepsilon })$ in the non-split case. Then the pull-back computation may be interpreted as expressing the pull-back of the Whittaker period on $\mathrm {GL}_4$ as an integral involving the Bessel period on $\mathrm {GSO}_{3,3}$, which is given in [FM22]. Thus, as in the non-split case, we are reduced to the Ichino–Ikeda-type explicit formula for the Whittaker period on $\mathrm {GL}_4$.

We now give the statement of the theorem in the split case.

Theorem 1.3 Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G(\mathbb {A})$ with trivial central character. Suppose that $D$ is split and the Arthur parameter of $\pi$ is generic.

Let $\xi \in D^-(F)$ such that $F(\xi )\simeq F\oplus F$ and fix an $F$-isomorphism $T_\xi \simeq F^\times \times F^\times$. For a character $\Lambda$ of $\mathbb {A}^\times / F^\times$, we also denote by $\Lambda$ the character of $T_\xi (\mathbb {A})$ defined by $\Lambda (a,b):=\Lambda (ab^{-1})$.

The following assertions hold.

1. (1) The $(\xi, \Lambda,\psi )$-Bessel period does not vanish on $V_\pi$ if and only if $\pi$ is generic and $L(\frac {1}{2},\pi \times \Lambda )\ne 0$. Here we note that $L(\frac {1}{2},\pi \times \Lambda ^{-1})$ is the complex conjugate of $L(\frac {1}{2},\pi \times \Lambda )$ since $\pi$ is self-dual.

2. (2) Further assume that $\pi$ is tempered. Then for any non-zero decomposable cusp form $\phi =\otimes _v\,\phi _v\in V_\pi$, we have

   \begin{align*} \frac{|B_{\xi, \Lambda,\psi}(\phi)|^2}{( \phi,\phi )_\pi} &= 2^{-\ell(\pi)}\,C_{\xi} \cdot \bigg(\prod_{j=1}^2\zeta_F(2j)\bigg)\frac{L(\frac{1}{2},\pi \times \Lambda)L(\frac{1}{2}, \pi \times \Lambda^{-1})}{L(1,\pi,\mathrm{Ad})\zeta_F(1)} \cdot\prod_v\frac{\alpha_v^\natural(\phi_v)}{(\phi_v,\phi_v)_{\pi_v}} \end{align*}
   where $\zeta _F(1)$ stands for $\mathrm {Res}_{s=1}\, \zeta _F(s)$.

1.8 Generalized Böcherer conjecture

Thanks to the meticulous local computation by Dickson, Pitale, Saha and Schmidt [DPSS20], Theorem 1.2 implies the generalized Böcherer conjecture. For brevity, we only state the scalar-valued full modular case here in the introduction. Indeed, a more general version shall be proved in § 8.3 as Theorem 8.1.

Theorem 1.4 Let $\Phi$ be a holomorphic Siegel cusp form of degree two and weight $k$ with respect to $\mathrm {Sp}_2(\mathbb {Z})$ which is a Hecke eigenform and $\pi (\Phi )$ the associated automorphic representation of $\mathbb {G}(\mathbb {A}_\mathbb {Q})$. Let

(1.8.1)\begin{equation} \Phi(Z)=\sum_{T>0}a(\Phi, T)\exp[2\pi\sqrt{-1}\operatorname{tr}(TZ)],\quad Z\in\mathfrak H_2, \end{equation}

be the Fourier expansion of $\Phi$ where $T$ runs over semi-integral positive-definite two-by-two symmetric matrices and $\mathfrak H_2$ denotes the Siegel upper half-space of degree two.

Let $E$ be an imaginary quadratic extension of $\mathbb {Q}$. We denote by $-D_E$ its discriminant, $\mathrm {Cl}_E$ its ideal class group and $w(E)$ the number of distinct roots of unity in $E$. In (1.8.1), when $T^\prime ={}^t\gamma T\gamma$ for some $\gamma \in \mathrm {SL}_2(\mathbb {Z})$, we have $a(\Phi, T^\prime )=a(\Phi,T)$. By the Gauss composition law, we may naturally identify the $\mathrm {SL}_2(\mathbb {Z})$-equivalence classes of binary quadratic forms of discriminant $-D_E$ with the elements of $\mathrm {Cl}_E$. Thus, the notation $a(\Phi, c)$ for $c \in \mathrm {Cl}_E$ makes sense. For a character $\Lambda$ of $\mathrm {Cl}_E$, we define $\mathcal {B}_\Lambda (\Phi,E)$ by

\[ \mathcal{B}_\Lambda(\Phi, E): = w(E)^{-1} \cdot \sum_{c \in \mathrm{Cl}_E} a(\Phi, c) \Lambda^{-1}(c).\label{mathcal B Phi E} \]

Suppose that $\Phi$ is not a Saito–Kurokawa lift. Then we have

(1.8.2)\begin{equation} \frac{|\mathcal{B}_\Lambda(\Phi, E)|^2}{\langle \Phi, \Phi \rangle} =2^{2k-4} \cdot D_E^{k-1} \cdot \frac{L(\frac{1}{2},\pi (\Phi) \times \mathcal{AI} (\Lambda))}{L(1,\pi (\Phi),\mathrm{Ad})}. \end{equation}

Here

\[ \langle\Phi,\Phi\rangle=\int_{\mathrm{Sp}_2(\mathbb{Z}) \backslash \mathfrak H_2} |\Phi(Z)|^2\det(Y)^{k-3}\,dX\, dY\quad \text{where $Z=X+\sqrt{-1}\,Y$.} \]

Remark 1.11 In Theorem 8.1, we prove (1.8.2) allowing $\Phi$ to have a square-free level and to be vector-valued. Moreover, assuming the temperedness of $\pi (\Phi )$, the weight $2$ case, which is of significant interest because of the modularity conjecture for abelian surfaces, is also included.

Formula (1.8.2) and its generalization (8.3.1) are expected to have a broad spectrum of interesting applications both arithmetic and analytic. Some of the examples are [Blo19], [DPSS20, § 3], [Dum22], [HY24], [Sah14] and [Wai19].

1.9 Organization of the paper

This paper is organized as follows. In § 2, we introduce some more notation and define local and global Bessel periods. In § 3, we carry out the pull-back computation of Bessel periods. In § 4, we shall prove Theorem 1.1 using the results in § 3. We also note some consequences of our proof of Theorem 1.1(i), which will be used in the proof of Theorem 1.2 later. In § 5, we recall the Rallis inner product formula for similitude groups. In § 6, we will give an explicit formula for Bessel periods on $\mathrm {GU}_{4,\varepsilon }$ in certain cases as explained in our strategy for the proof of Theorem 1.2. In § 7, we complete our proof of Theorem 1.2. In § 8, we prove the generalized Böcherer conjecture, including the vector valued case. In Appendix A, we will give an explicit formula of Whittaker periods for irreducible cuspidal tempered automorphic representations of $G$. In Appendix B, we compute the local Bessel periods explicitly for representation of $G(\mathbb {R})$ corresponding to vector-valued holomorphic Siegel modular forms. This result is used in § 8. In Appendix C, we consider the meromorphic continuation of the $L$-function for $\mathrm {SO}(5) \times \mathrm {SO}(2)$.

1.10 Index of notation

1. $M_D, N_D$, 2117

2. $G_D$, 2116

3. $G$, 2117

4. $G_D^1$, 2117

5. $G^1$, 2117

6. $T_{\xi }$, 2117

7. $G_D^+$, 2118

8. $B_{\xi, \Lambda, \psi }$, 2118

9. $\mathcal {AI}(\Lambda )$, 2119

10. $\mathcal {B}_\Lambda (\Phi, E)$, 2125

11. $J_m$, 2126

12. $\mathrm {GO}_{n+2, n}, \mathrm {GSO}_{n+2, n}$, 2126

13. $\mathrm {GU}_{3, D}, \mathrm {GSU}_{3, D}$, 2127

14. $\mathrm {GU}_{4, \varepsilon }$, 2127

15. $\Phi _D$, 2129

16. $\Phi$, 2130

17. $M$, $N$, 2130

18. $T_S$, 2130

19. $B_{S, \Lambda, \psi }$, 2131

20. $M_{3, D}$ $N_{3, D}$, 2131

21. $M_{X, D}$, 2131

22. $\mathcal {B}_{X, \chi, \psi }$, 2131

23. $\mathcal {B}_{X, \chi, \psi }^D$, 2131

24. $M_{4,2}$ $N_{4,2}$, 2132

25. $M_{X}$, 2133

26. $\alpha _{\chi, \psi _N}(\phi, \phi ^\prime )$, 2133

27. Type I-A, Type I-B, 2152

28. $W^{\psi _U}$, 2161

29. $\mathcal {W}^{\psi _U}$, 2165

30. $\mathcal {W}^\circ _{G, v}, \mathcal {W}_{G, v}$, 2163

31. $\mathcal {L}_v^\circ (\phi _v, f_v), \mathcal {L}_v(\phi _v, f_v)$, 2164

32. $W_{\psi _{U_G}}$, 2163

33. $\phi _{\Phi, S}$, 2193

2. Preliminaries

2.1 Groups

2.1.1 Quaternion algebras

Let $X(E : F)$ denote the set of $F$-isomorphism classes of central simple algebras over $F$ containing $E$. Then we recall that the map $\varepsilon \mapsto D_{\varepsilon }$ gives a bijection between $F^\times / \mathrm {N}_{E / F}(E^\times )$ and $X(E : F)$ (see [FS03, Lemma 1.3]) where

(2.1.1) \begin{equation} D_\varepsilon := \bigg\{\!\begin{pmatrix} a & \varepsilon b\\ b^\sigma & a^\sigma \end{pmatrix} : a, b \in E \bigg\}\quad\text{for $\varepsilon\in F^\times$}. \end{equation}

Here we regard $E$ as a subalgebra of $D_\varepsilon$ by

\[ E \ni a \mapsto \begin{pmatrix}a & 0\\ 0 & a^\sigma \end{pmatrix} \in D_\varepsilon. \]

We also note that $D_\varepsilon \simeq \mathrm {Mat}_{2 \times 2}(F)$ when $\varepsilon \in \mathrm {N}_{E/ F}(E^\times )$. The canonical involution $D_\varepsilon \ni x\mapsto \bar {x}\in D_\varepsilon$ is given by

\[ \bar{x}=\begin{pmatrix} a^\sigma & -\varepsilon b\\ -b^\sigma & a\end{pmatrix}\quad\text{for $x=\begin{pmatrix}a & \varepsilon b\\ b^\sigma & a^\sigma\end{pmatrix}$}. \]

We denote the reduced trace of $D$ by $\mathrm {tr}_D$.

2.1.2 Orthogonal groups

For a non-negative integer $n$, a symmetric matrix $S_n\in \mathrm {Mat}_{(2n+2) \times (2n+2)}(F)$ is defined inductively by

(2.1.2)\begin{equation} S_0: = \begin{pmatrix}2 & 0\\ 0 & -2d \end{pmatrix}\quad \text{and} \quad S_n: = \begin{pmatrix}0 & 0 & 1\\ 0 & S_{n-1} & 0\\ 1 & 0 & 0\end{pmatrix}\quad \text{for $n\ge 1$}. \end{equation}

We recall that $E=F(\eta )$ where $\eta ^2=d$. Then we denote the corresponding orthogonal group, the special orthogonal group and the similitude orthogonal group by

(2.1.3)\begin{equation} \mathrm{O}(S_n)=\mathrm{O}_{n+2,n},\quad \mathrm{SO}(S_n)= \mathrm{SO}_{n+2,n}\quad \text{and}\quad \mathrm{GO}(S_n)=\mathrm{GO}_{n+2,n}, \end{equation}

respectively. Let $\mathrm {GSO}_{n+2,n}$ denote the identity component of $\mathrm {GO}_{n+2,n}$. Thus,

(2.1.4)\begin{equation} \mathrm{GSO}_{n+2,n}(F) = \{g \in \mathrm{GO}_{n+2,n}(F): \det (g) = \lambda(g)^{n+1}\}, \end{equation}

where

(2.1.5)\begin{equation} \mathrm{GO}_{n+2,n}(F) = \{g \in \mathrm{GL}_{2n+2}(F) :{}^{t}g\, S_{n} \,g = \lambda(g)S_n,\ \lambda(g) \in F^\times\}. \end{equation}

For a positive integer $n$, we denote by $J_{2n}$ the $2n\times 2n$ symmetric matrix with ones on the non-principal diagonal and zeros elsewhere, i.e.

(2.1.6)\begin{equation} J_2=\begin{pmatrix}0 & 1\\1 & 0\end{pmatrix}\quad\text{and}\quad J_{2(n+1)}=\begin{pmatrix}0 & 0 & 1\\0 & J_{2n} & 0\\1 & 0 & 0\end{pmatrix}\quad\text{for $n\ge 1$}. \end{equation}

Then the similitude orthogonal group $\mathrm {GO}_{n,n}$ is defined by

(2.1.7)\begin{equation} \mathrm{GO}_{n,n}(F):=\{g \in\mathrm{GL}_{2n}(F): {}^{t}g\, J_{2n}\, g = \lambda(g) J_{2n}, \lambda(g)\in F^\times\} \end{equation}

and we denote by $\mathrm {GSO}_{n,n}$ its identity component, which is given by

(2.1.8)\begin{equation} \mathrm{GSO}_{n,n}(F) = \{g \in \mathrm{GO}_{n,n}(F) : \det (g) = \lambda(g)^{n}\}. \end{equation}

2.1.3 Quaternionic unitary groups

Let $D$ be a quaternion algebra over $F$ containing $E$. Recall that $G_D$ denotes the similitude quaternionic unitary group of degree $2$ defined by (1.4.2).

We define a similitude quaternionic unitary group $\mathrm {GU}_{3,D}$ of degree $3$ by

(2.1.9)\begin{equation} \mathrm{GU}_{3,D}(F):= \{g \in \mathrm{GL}_3(D) : {}^t \bar{g} \,{\bf J}_\eta\, g = \lambda(g) {\bf J}_\eta,\ \lambda(g)\in F^\times\}, \end{equation}

where we define a skew-hermitian matrix ${\bf J}_\eta$ by

(2.1.10)\begin{equation} {\bf J}_\eta := \begin{pmatrix} 0 & 0 & \eta\\0 & \eta & 0\\ \eta & 0 & 0\end{pmatrix}. \end{equation}

Here $\bar {A}=(\bar {a}_{ij})$ for $A=(a_{ij})\in \mathrm {Mat}_{m \times n}(D)$. Let us denote by $\mathrm {GSU}_{3,D}$ the identity component of $\mathrm {GU}_{3,D}$. Then unlike the orthogonal case, as noted in [MVW87, p. 21–22], we have

\[ \mathrm{GSU}_{3,D}(F) = \mathrm{GU}_{3,D}(F) \]

and

\[ \text{$\mathrm{GSU}_{3,D}(F_v) = \mathrm{GU}_{3,D}(F_v)$ when $D \otimes_F F_v$ is not split.} \]

Moreover, when $D\otimes _F F_v$ is split at a place $v$ of $F$, we have

(2.1.11)\begin{equation} \mathrm{GU}_{3, D}(F_v)\simeq \begin{cases} \mathrm{GO}_{4,2}(F_v) & \text{if $E\otimes F_v$ is a quadratic extension of $F_v$};\\ \mathrm{GO}_{3,3}(F_v) & \text{if $E\otimes F_v\simeq F_v\oplus F_v$.} \end{cases} \end{equation}

We also define $\mathrm {GU}_{1,D}$ by

(2.1.12)\begin{equation} \mathrm{GU}_{1,D}(F):= \{\alpha\in D^\times :\bar{\alpha} \eta \alpha = \lambda(\alpha) \eta,\ \lambda(\alpha)\in F^\times\} \end{equation}

and denote its identity component by $\mathrm {GSU}_{1,D}$. Then we note that

(2.1.13)\begin{align} \mathrm{GSU}_{1,D}(F)&=\{\alpha\in D^\times:\bar{\alpha} \eta \alpha = \mathrm{n}_D(\alpha) \eta\} \nonumber\\ &=\{x\in D^\times\mid x\eta=\eta x\}=T_\eta , \end{align}

where $T_\eta$ is defined by (1.4.6) with $\xi =\eta$ and $\mathrm {n}_D$ denotes the reduced norm of $D$.

2.1.4 Unitary groups

Suppose that $D=D_\varepsilon$ defined by (2.1.1). Then we define $\mathrm {GU}_{4,\varepsilon }$ a similitude unitary group of degree $4$ by

(2.1.14)\begin{equation} \mathrm{GU}_{4,\varepsilon}(F):=\{g \in \mathrm{GL}_4(E) : {}^{t}g^\sigma \mathcal{J}_\varepsilon g = \lambda(g) \mathcal{J}_\varepsilon,\ \lambda(g)\in F^\times\}, \end{equation}

where we define a hermitian matrix $\mathcal {J}_\varepsilon$ by

\[ \mathcal{J}_\varepsilon := \begin{pmatrix} 0 & 0 & 0 & 1\\ 0 & -1 & 0 & 0\\ 0 & 0 & \varepsilon & 0\\ 1 & 0 & 0 & 0\end{pmatrix}. \]

Here $A^\sigma =(a_{ij}^\sigma )$ for $A=(a_{ij})\in \mathrm {Mat}_{m \times n}(E)$. Then we have

(2.1.15)\begin{equation} \mathrm{GU}_{4,\varepsilon}\simeq \begin{cases} \mathrm{GU}_{2,2}, & \text{when $D$ is split, i.e. $\varepsilon\in \mathrm{N}_{E/ F}(E^\times)$};\\ \mathrm{GU}_{3,1}, & \text{when $D$ is non-split, i.e. $\varepsilon\notin \mathrm{N}_{E/ F}(E^\times)$}. \end{cases} \end{equation}

We also define $\mathrm {GU}_{2,\varepsilon }$ a similitude unitary group of degree $2$ by

(2.1.16)\begin{equation} \mathrm{GU}_{2,\varepsilon}(F) := \{ g \in \mathrm{GL}_{2}(E) : {}^{t}g^\sigma J_\varepsilon g =\lambda(g) J_\varepsilon,\ \lambda(g) \in F^\times\}\quad \text{where}\ J_\varepsilon=\begin{pmatrix}-1 & 0\\0 & \varepsilon \end{pmatrix}. \end{equation}

2.2 Accidental isomorphisms

We need to explicate the accidental isomorphisms of our concern, since we use them in a crucial way to transfer an automorphic period on one group to the one on the other group. The reader may consult, for example, Satake [Sat61] and Tsukamoto [Tsu61] about the details of the material here.

2.2.1 $\mathrm {PGSU}_{3,D}\simeq \mathrm {PGU}_{4,\varepsilon }$

Suppose that $D=D_\varepsilon$. Then we may naturally realize $\mathrm {GSU}_{3,D}(F)$ as a subgroup of $\mathrm {GL}_6(E)$. We note that

\[ \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0\\ 0 & -\varepsilon & 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 0 & 0\\ 0 & 0 & 0 & -\varepsilon & 0 & 0\\ 0 & 0 & 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 0 & 0 & -\varepsilon \end{pmatrix}\,{}^{t}\bar{g}\, \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0\\ 0 & -\varepsilon & 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 0 & 0\\ 0 & 0 & 0 & -\varepsilon & 0 & 0\\ 0 & 0 & 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 0 & 0 & -\varepsilon \end{pmatrix}^{-1}= {}^{t}g^\sigma \]

and

\[ \begin{pmatrix} 0 & 1 & 0 & 0 & 0 & 0\\ -1 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 1 & 0 & 0\\ 0 & 0 & -1 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0 & -1 & 0 \end{pmatrix}\,{}^{t}\bar{g}\, \begin{pmatrix} 0 & 1 & 0 & 0 & 0 & 0\\ -1 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 1 & 0 & 0\\ 0 & 0 & -1 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0 & -1 & 0 \end{pmatrix}^{-1}= {}^{t}g. \]

Thus, in this realization, we have

(2.2.1)\begin{equation} \mathrm{GSU}_{3,D}(F) = \{ g \in \mathrm{GSO}_{3,3}(E) : {}^{t} g^\sigma\, \mathcal{J}_\varepsilon^\circ \,g =\lambda(g)\mathcal{J}_{\varepsilon}^\circ,\ \lambda(g)\in F^\times \}, \end{equation}

where

\[ \mathcal{J}_{\varepsilon}^\circ =- \begin{pmatrix} 0 & 0 & 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 0 & 0 & \varepsilon\\ 0 & 0 & 1 & 0 & 0 & 0\\ 0 & 0 & 0 & \varepsilon & 0 & 0\\ 1 & 0 & 0 & 0 & 0 & 0\\ 0 & \varepsilon & 0 & 0 & 0 & 0 \end{pmatrix}. \]

Here we recall that

(2.2.2)\begin{equation} \mathrm{GSO}_{3,3}(E) \simeq \mathrm{GL}_4(E) \times \mathrm{GL}_1(E) / \{ (z, z^{-2} ) :z \in E^\times \}. \end{equation}

In fact, the isomorphism (2.2.2) is realized as follows. Let us take the standard basis

\[ b_1 ={}^{t}(1,0,0,0),\quad b_2 ={}^{t}(0,1,0,0),\quad b_3 ={}^{t}(0,0,1,0),\quad b_4 ={}^{t}(0,0,0,1), \]

of $E^4$. Then we may consider $V:=\wedge ^2 E^4$ as an orthogonal space over $E$ with a quadratic form $(\,{,}\,)_V$ defined by

\[ v_1\wedge v_2=(v_1 ,v_2)_V \cdot b_1 \wedge b_2 \wedge b_3 \wedge b_4 \]

for $v_1, v_2 \in V$. As a basis of $V$ over $E$, we take $\{ \varepsilon _i: 1\le i\le 6 \}$ given by

\[ \varepsilon_1 =b_1 \wedge b_2,\quad \varepsilon_2 =b_1 \wedge b_3,\quad \varepsilon_3 = b_1 \wedge b_4,\quad \varepsilon_4 = b_2 \wedge b_3,\quad \varepsilon_5 = b_4 \wedge b_2,\quad \varepsilon_6 = b_3 \wedge b_4. \]

Let the group $\mathrm {GL}_4(E) \times \mathrm {GL}_1(E)$ act on $V$ by $(g, a)(w_1 \wedge w_2) = a \cdot (gw_1 \wedge gw_2)$ where $w_1,w_2\in E^4$. This action defines a homomorphism

(2.2.3)\begin{equation} \mathrm{GL}_4(E) \times \mathrm{GL}_1(E)\to \mathrm{GSO}_{3,3}(E), \end{equation}

where we take $\{\varepsilon _i: 1\le i\le 6\}$ as a basis of $V$ and the homomorphism (2.2.3) induces the isomorphism (2.2.2). By a direct computation we observe that $(-\mathcal {J}_\varepsilon, 1)$ is mapped to $\mathcal {J}_\varepsilon ^\circ$ under (2.2.3) and the restriction of the homomorphism (2.2.3) gives a homomorphism

(2.2.4)\begin{equation} \mathrm{GU}_{4,\varepsilon}(F)\to \mathrm{GSU}_{3,D}(F). \end{equation}

Then it is easily seen that the isomorphism

(2.2.5)\begin{equation} \Phi_D : \mathrm{PGU}_{4,\varepsilon}(F)\overset{\sim}{\to} \mathrm{PGSU}_{3,D}(F) \end{equation}

is induced.

2.2.2 $\mathrm {PGU}_{2,2} \simeq \mathrm {PGSO}_{4,2}$

When $\varepsilon \in \mathrm {N}_{E / F}(E^\times )$, the quaternion algebra $D=D_\varepsilon$ is split and the isomorphism (2.2.5) gives an isomorphism $\mathrm {PGU}_{2,2}\simeq \mathrm {PGSO}_{4,2}$. We recall the concrete realization of this isomorphism. First we define $\mathrm {GU}_{2,2}$ by

\begin{align*} \mathrm{GU}_{2,2} &:= \{g\in\mathrm{GL}_4(E): {}^tg^\sigma\, J_4\,g =\lambda(g)J_4, \lambda(g)\in F^\times\}, \quad \text{where $J_4=\begin{pmatrix} 0 & 0 & 0 & 1\\ 0 & 0 & 1 & 0\\ 0 & 1 & 0 & 0\\ 1 & 0 & 0 & 0 \end{pmatrix}$} \end{align*}

as (2.1.6). Let

\[ {\mathcal{V}}: = \left\{B((x_i)_{1\le i\le 6}):= \left(\begin{matrix} 0 & \eta x_1 & x_3 + \eta x_4 & x_2\\ -\eta x_1 & 0 & x_5 & -x_3 +\eta x_4\\ -x_3 -\eta x_4 & -x_5 & 0 & \eta^{-1} x_6\\ -x_2 & x_3 -\eta x_4 & -\eta^{-1} x_6 & 0\\ \end{matrix}\right): x_i \in F (1\le i\le6)\right\}. \]

We define $\Psi : {\mathcal {V}} \rightarrow F$ by

\[ \Psi (B): = {\rm Tr} \bigg(B\, \begin{pmatrix} 0 & 1_2\\ 1_2 & 0\end{pmatrix} \,{}^{t}B^\sigma\, \begin{pmatrix}0 & 1_2\\ 1_2 & 0\end{pmatrix}\bigg). \]

Then we have

\[ \Psi(B((x_i)_{1\le i\le 6}))=-4\{x_1x_6+x_2x_5-(x_3^2-dx_4^2)\}. \]

Let $\mathrm {GSU}_{2,2}$ denote the identity component of $\mathrm {GU}_{2,2}$, i.e.

\[ \mathrm{GSU}_{2,2} = \{ g \in \mathrm{GU}_{2,2}: \det(g) = \lambda(g)^2 \}. \]

We let $\mathrm {GSU}_{2,2}$ act on $\mathcal {V}$ by

\[ \mathrm{GSU}_{2,2}\times \mathcal{V}\ni (g,B)\mapsto (wg w)B(w\,{}^tg w)\in\mathcal{V}, \quad\text{where $w=\begin{pmatrix} 1 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1\\ 0 & 0 & 1 & 0\end{pmatrix}$.} \]

Then this action induces a homomorphism $\phi : \mathrm {GSU}_{2,2} \rightarrow \mathrm {GO}({\mathcal {V}})$. We note that

\[ \lambda(\phi(g)) = \det(g)\quad \text{for $g\in \mathrm{GSU}_{2,2}$} \]

and this implies that the image of $\phi$ is contained in $\mathrm {GSO}(\mathcal {V})$. As a basis of ${\mathcal {V}}$, we may take

\begin{gather*} f_1=\begin{pmatrix} 0 & \eta & 0 & 0\\ -\eta & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ \end{pmatrix},\quad f_2=\begin{pmatrix} 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ -1 & 0 & 0 & 0\\ \end{pmatrix},\quad f_3=\begin{pmatrix} 0 & 0 & 1 & 0\\ 0 & 0 & 0 & -1\\ -1 & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ \end{pmatrix},\\ f_4= \begin{pmatrix} 0 & 0 & \eta & 0\\ 0 & 0 & 0 & \eta\\ -\eta & 0 & 0 & 0\\ 0 & -\eta & 0 & 0\\ \end{pmatrix},\quad f_5=\begin{pmatrix} 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & -1 & 0 & 0\\ 0 & 0 & 0 & 0\\ \end{pmatrix},\quad f_6=\begin{pmatrix} 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & \eta^{-1}\\ 0 & 0 & -\eta^{-1} & 0\\ \end{pmatrix}. \end{gather*}

With respect to this basis, we may regard $\phi$ as a homomorphism from $\mathrm {GSU}_{2,2}$ to $\mathrm {GO}_{4,2}$, where the group $\mathrm {GO}_{4,2}$ is given by (2.1.5) for $n=2$. Let us consider $\mathrm {GSU}_{2,2}\rtimes E^\times$ where the action of $\alpha \in E^\times$ on $g\in \mathrm {GSU}_{2,2}$ is given by

\[ \alpha \cdot g = \begin{pmatrix} \alpha & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & (\alpha^{\sigma})^{-1} \end{pmatrix}\,g \,\begin{pmatrix} \alpha & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & (\alpha^{\sigma})^{-1} \end{pmatrix}^{-1}. \]

Then as in [Mor14a, p. 32–34], $\phi$ may be extended to $\mathrm {GSU}_{2,2}\rtimes E^\times$ and we have a homomorphism $\mathrm {GSU}_{2,2}\rtimes E^\times \to \mathrm {PGSO}_{4,2}$ which induces the isomorphism

(2.2.6)\begin{equation} \Phi: \mathrm{PGU}_{2,2} \overset{\sim}{\to} \mathrm{PGSO}_{4,2}. \end{equation}

2.3 Bessel periods

Let us introduce Bessel periods on various groups.

2.3.1 Bessel periods on $G=\mathrm {GSp}_2$

Though we already introduced Bessel periods on $G_D$ in general as (1.4.11), we would like to describe them concretely in the case of $G$ here for our explicit pull-back computations in the next section.

Let $P$ be the Siegel parabolic subgroup of $G$ with the Levi decomposition $P = MN$ where

\[ M(F) = \bigg\{\!\begin{pmatrix} g & 0\\ 0 & \lambda \cdot {}^tg^{-1} \end{pmatrix}: \begin{aligned} & g \in \mathrm{GL}_2(F),\\ & \lambda \in F^\times \end{aligned}\bigg\},\quad N(F) = \bigg\{\!\begin{pmatrix} 1 & X\\ 0 & 1\end{pmatrix} : X \in \mathrm{Sym}_2(F)\bigg\}.\label{d:N} \]

Here $\mathrm {Sym}_n(F)$ denotes the set of $n$ by $n$ symmetric matrices with entries in $F$ for a positive integer $n$. For $S \in \mathrm {Sym}_2(F)$, let us define a character $\psi _{S}$ of $N(\mathbb {A})$ by

\[ \psi_{S} \begin{pmatrix} 1 & X\\ 0 & 1 \end{pmatrix} = \psi[\mathrm{tr}(SX)]. \]

For $S\in \mathrm {Sym}_2(F)$ such that $\det S\ne 0$, let

\[ T_S := \{ g \in\mathrm{GL}_2 : {}^{t}gSg = \det(g)S\}.\label{T_S} \]

We identify $T_S$ with the subgroup of $G$ given by

\[ \bigg\{\!\begin{pmatrix} g & 0\\ 0 & \det (g) \cdot {}^{t}g^{-1} \end{pmatrix} : g \in T_S\bigg\}. \]

Definition 2.1 Let us take $S\in \mathrm {Sym}_2(F)$ such that $T_S(F)$ is isomorphic to $E^\times$. Let $\pi$ be an irreducible cuspidal automorphic representation of $G(\mathbb {A})$ whose central character is trivial and $V_\pi$ its space of automorphic forms. Fix an $F$-isomorphism $T_S(F)\simeq E^\times$. Let $\Lambda$ be a character of $\mathbb {A}_E^\times / E^\times$ such that $\Lambda \mid _{\mathbb {A}^\times }$ is trivial. We regard $\Lambda$ as a character of $T_S(\mathbb {A}) / \mathbb {A}^\times \, T_S(F)$.

Then for $\varphi \in V_\pi$, we define $B_{S,\Lambda,\psi }(\varphi )$, the $(S,\Lambda,\psi )$-Bessel period of $\varphi$ by

(2.3.1)\begin{equation} B_{S, \Lambda,\psi}(\varphi) = \int_{\mathbb{A}^\times \,T_{S}(F) \backslash T_{S}(\mathbb{A})} \int_{N(F) \backslash N(\mathbb{A})} \varphi(uh) \Lambda^{-1}(h) \psi_{S}^{-1}(u) \, du\, dh. \end{equation}

We say that $\pi$ has the $(S,\Lambda,\psi )$-Bessel period when $B_{S, \Lambda,\psi }\not \equiv 0$ on $V_\pi$. Then we also say that $\pi$ has the $(E,\Lambda )$-Bessel period as in Definition 1.2.

2.3.2 Bessel periods on $\mathrm {GSU}_{3,D}$

Let us introduce Bessel periods on the group $\mathrm {GSU}_{3,D}$ defined in § 2.1.3. Let $P_{3,D}$ be a maximal parabolic subgroup of $\mathrm {GSU}_{3,D}$ with the Levi decomposition $P_{3,D}=M_{3,D}N_{3,D}$ where

\[ M_{3,D} = \left\{\begin{pmatrix} g & 0 & 0\\ 0 & h & 0\\ 0 & 0 & g \end{pmatrix}:\begin{aligned} & g\in D^\times,\\ & h \in T_\eta,\\ & \mathrm{n}_D(g)=\mathrm{n}_D(h) \end{aligned}\right\},\quad N_{3,D} = \left\{\begin{pmatrix} 1 & A^\prime & B\\ 0 & 1 & A\\ 0 & 0 & 1 \end{pmatrix} \in \mathrm{GSU}_{3,D}\right\}.\label{d:N3,D} \]

As for $T_\eta$, we recall (2.1.13) and $T_\eta \simeq E^\times$. For $X \in D^\times$, we define a character $\psi _{X, D}$ of $N_{3, D}(\mathbb {A})$ by

\[ \psi_{X, D} \begin{pmatrix} 1 & A^\prime & B\\ 0 & 1 & A\\ 0 & 0 & 1 \end{pmatrix} = \psi [\mathrm{tr}_D(XA)]. \]

Then the identity component of the stabilizer of $\psi _{X, D}$ in $M_{3,D}$ is

\[ M_{X, D} = \left\{\!\begin{pmatrix} h^{X} & 0 & 0\\ 0 & h & 0\\ 0 & 0 & h^{X} \end{pmatrix}: h \in T_\eta \right\},\quad \text{where}\ h^X = XhX^{-1}.\label{M_X D} \]

We identify $M_X$ with $T_\eta$ by

(2.3.2)\begin{equation} M_{X, D}\ni \begin{pmatrix} h^{X} & 0 & 0\\ 0 & h & 0\\ 0 & 0 & h^{X}\end{pmatrix} \mapsto h\in T_\eta \end{equation}

and we fix an $F$-isomorphism $T_\eta \simeq E^\times$.

Definition 2.2 Let $\sigma _D$ be an irreducible cuspidal automorphic representation of $\mathrm {GSU}_{3,D}(\mathbb {A})$ and $V_{\sigma _D}$ its space of automorphic forms. Let $\chi$ be a character of $\mathbb {A}_E^\times / E^\times$ and we regard $\chi$ as a character of $M_{X,D}(\mathbb {A}) / M_{X,D}(F)$. Suppose that $\chi |_{\mathbb {A}^\times } = \omega _{\sigma _D}$, the central character of $\sigma _D$.

Then for $\varphi \in V_{\sigma _D}$, we define $\mathcal {B}^D_{X, \chi, \psi } (\varphi )$, the $(X,\chi, \psi )$-Bessel period of $\varphi$ by

(2.3.3) \begin{align} \mathcal{B}^D_{X, \chi, \psi} (\varphi) &= \int_{\mathbb{A}^\times M_{X,D}(F) \backslash M_{X,D}(\mathbb{A})} \int_{N_{3,D}(F) \backslash N_{3,D}(\mathbb{A})} \varphi(uh) \chi (h)^{-1} \psi_{X, D}(u)^{-1} \, du \, dh. \end{align}

2.3.3 Bessel periods on $\mathrm {GU}_{4,\varepsilon }$

In light of the accidental isomorphism (2.2.5), Bessel periods on the group $\mathrm {GU}_{4,\varepsilon }$ is defined as follows.

Let $P_{4,\varepsilon }$ be a maximal parabolic subgroup of $\mathrm {GU}_{4,\varepsilon }$ with the Levi decomposition $M_{4,\varepsilon }N_{4,\varepsilon }$ where

\begin{align*} M_{4,\varepsilon}(F) &= \left\{\!\begin{pmatrix} a & 0 & 0\\ 0 & g & 0\\ 0 & 0 & \lambda(g) (a^\sigma)^{-1} \end{pmatrix}: a \in E^\times, g \in \mathrm{GU}_{2,\varepsilon}(F) \right\},\\ N_{4,\varepsilon}(F) &= \left\{\!\begin{pmatrix} 1 & A & B\\ 0 & 1_2 & A^\prime\\ 0 & 0 & 1 \end{pmatrix} \in \mathrm{GU}_{4,\varepsilon}(F) \right\}. \end{align*}

Let us take an anisotropic vector $e\in E^4$ of the form ${}^{t}(0, \ast, \ast, 0)$. Then we define a character $\chi _e$ of $N_{4,\varepsilon }(\mathbb {A})$ by

\[ \chi_e (u)= \psi( (ue, b_1)_\varepsilon),\quad \text{where $(x,y)_\varepsilon={}^tx^\sigma J_{\varepsilon}y$}. \]

Here we recall that $J_\varepsilon$ is as given in (2.1.16) and $b_1={}^t(1,0,0,0)$. Let $D_e$ denote the subgroup of $M_{4,\varepsilon }$ given by

\[ D_{e} := \left\{\!\begin{pmatrix}1 & 0 & 0\\ 0 & h & 0\\ 0 & 0 & 1\end{pmatrix}: h \in \mathrm{U}_{2,\varepsilon},\ h e = e \right\}. \]

Then the group $D_e(\mathbb {A})$ stabilizes the character $\chi _e$ by conjugation. We note that

\[ D_e(F) \simeq \mathrm{U}_1(F):= \{ a \in E^\times : \bar{a}a =1 \}. \]

Hence, for a character $\Lambda$ of $\mathbb {A}_E^\times$ which is trivial on $\mathbb {A}^\times$, we may regard $\Lambda$ as a character of $D_e(\mathbb {A})$ by $d \mapsto \Lambda (\det d)$. Then we define a character $\chi _{e, \Lambda }$ of $R_e(\mathbb {A})$ where $R_e:=D_eN_{4,\varepsilon }$ by

(2.3.4)\begin{equation} \chi_{e, \Lambda}(ts) := \Lambda(t) \chi_e(s)\quad \text{for}\ t \in D_e(\mathbb{A}),\enspace s \in N_{4,\varepsilon}(\mathbb{A}). \end{equation}

Definition 2.3 For a cusp form $\varphi$ on $\mathrm {GU}_{4,\varepsilon }(\mathbb {A}_F)$ with a trivial central character, we define $B_{e, \Lambda,\psi }(\varphi )$, the $(e, \Lambda,\psi )$-Bessel period of $\varphi$, by

(2.3.5)\begin{equation} B_{e, \Lambda,\psi}(\varphi) = \int_{D_e(F) \backslash D_e(\mathbb{A}_F)} \int_{N_{4,\varepsilon}(F) \backslash N_{4,\varepsilon}(\mathbb{A}_F)} \chi_{e, \Lambda}(ts)^{-1}\, \varphi(ts) \, ds \, dt. \end{equation}

2.3.4 Bessel periods on $\mathrm {GSO}_{4,2}$ and $\mathrm {GSO}_{3,3}$

By combining the accidental isomorphisms (2.2.5) and (2.2.6) in the split case, we shall define Bessel periods on $\mathrm {GSO}_{4,2}$ and $\mathrm {GSO}_{3,3}$ as the following.

Let $P_{4,2}$ denote a maximal parabolic subgroup of $\mathrm {GSO}_{4,2}$ with the Levi decomposition $P_{4,2}= M_{4,2}N_{4,2}$ where

\[ M_{4,2} = \left\{\!\begin{pmatrix} g & 0 & 0\\0 & h & 0\\0 & 0 & g^\ast\cdot \det h\end{pmatrix} : \begin{aligned} & g \in \mathrm{GL}_2,\\ & h \in \mathrm{GSO}_{2 ,0} \end{aligned}\right\},\quad N_{4,2}= \left\{\!\begin{pmatrix} 1_2 & A^\prime & B\\ 0 & 1_2 & A\\ 0 & 0 & 1_2 \end{pmatrix} \in \mathrm{GSO}_{4,2}\right\}.\label{d:N4,2} \]

Here

\[ g^\ast = \begin{pmatrix}0 & 1\\1 & 0\end{pmatrix} {}^{t}g^{-1} \begin{pmatrix}0 & 1\\1 & 0\end{pmatrix}\quad\text{for}\ g\in\mathrm{GL}_2. \]

Then for $X\in \mathrm {Mat}_{2\times 2}(F)$, we define a character $\psi _X$ of $N_{4,2}(\mathbb {A})$ by

\[ \psi_{X} \begin{pmatrix}1_2 & A^\prime & B\\ 0 & 1_2 & A\\0 & 0 & 1_2 \end{pmatrix}= \psi [\mathrm{tr}(XA)]. \]

Suppose that $\det X \ne 0$ and let

\[ M_{X} := \left\{\!\begin{pmatrix} (\det h) \cdot (h^X)^\ast & 0 & 0\\ 0 & h & 0\\ 0 & 0 & h^X \end{pmatrix} : h \in \mathrm{GSO}_{2,0} \right\},\label{M_X} \]

where $h^X = XhX^{-1}$ . Then $M_X(\mathbb {A})$ stabilizes the character $\psi _X$ and $M_X$ is isomorphic to $\mathrm {GSO}_{2,0}$. We fix an isomorphism $\mathrm {GSO}_{2,0}(F)\simeq E^\times$ and we regard a character of $\mathbb {A}_E^\times$ as a character of $M_X(\mathbb {A})$.

Definition 2.4 Let $\sigma$ be an irreducible cuspidal automorphic representation of $\mathrm {GSO}_{4,2}(\mathbb {A})$ with its space of automorphic forms $V_\sigma$ and the central character $\omega _\sigma$. For a character $\chi$ of $\mathbb {A}_E^\times$ such that $\chi |_{\mathbb {A}^\times } = \omega _{\sigma }$, we define $\mathcal {B}_{X,\chi, \psi }(\varphi )$, the $(X, \chi, \psi )$-Bessel period of $\varphi \in V_\sigma$ by

(2.3.6)\begin{equation} \mathcal{B}_{X, \chi, \psi}(\varphi) = \int_{N_{4,2}(F) \backslash N_{4,2}(\mathbb{A})} \int_{M_{X}(F) \mathbb{A}^\times \backslash M_{X}(\mathbb{A})} \varphi(u h) \chi(h)^{-1} \psi_{X}(u)^{-1} \, du \, dh. \end{equation}

When $d \in (F^\times )^2$, we know that $\mathrm {GSO}(S_2) \simeq \mathrm {GSO}_{3,3}$. Hence, as above, for a cusp form $\varphi$ on $\mathrm {GSO}_{3,3}$ with central character $\omega$ and characters $\Lambda _1, \Lambda _2$ of $\mathbb {A}^\times / F^\times$ such that $\Lambda _1 \Lambda _2=\omega$, we define $(X, \Lambda _1, \Lambda _2, \psi )$-Bessel period by

\[ \mathcal{B}_{X, \Lambda, \psi}(\varphi) = \int_{N_{4,2}(F) \backslash N_{4,2}(\mathbb{A})} \int_{M_{X}(F) \mathbb{A}^\times \backslash M_{X}(\mathbb{A})} \varphi(u h) \chi_{\Lambda_1, \Lambda_2}(h)^{-1} \psi_{X}(u)^{-1} \, du \, dh. \]

Here, since $M_{4,2} \simeq \mathrm {GL}_2 \times \mathrm {GSO}_{1,1}$ and $\mathrm {GSO}_{1,1}(F) = \big \{\big (\begin{smallmatrix} a & \\ & b\end{smallmatrix}\big ) : a, b \in F^\times \big \}$, we define a character $\chi _{\Lambda _1, \Lambda _2}$ of $\mathrm {GSO}_{1,1}(\mathbb {A})$ by

\[ \chi_{\Lambda_1, \Lambda_2} \begin{pmatrix} a & \\ & b\end{pmatrix} =\Lambda_1(a) \Lambda_2(b). \]

When $\omega$ is trivial, we have $\Lambda _2 = \Lambda _1^{-1}$. In this case, we simply call $(X, \Lambda _1, \Lambda _1^{-1}, \psi )$-Bessel period as $(X, \Lambda _1, \psi )$-Bessel period and simply write $\chi _{\Lambda _1, \Lambda _1^{-1}} = \Lambda _1$.

2.4 Local Bessel periods

Let us introduce local counterparts to the global Bessel periods. Let $k$ be a local field of characteristic zero and $D$ a quaternion algebra over $k$.

Since the local Bessel periods are deduced from the global ones in a uniform way, by abuse of notation, let a quintuple $(H, T, N,\chi,\psi _N)$ stand for one of

\begin{align*} &\text{$(G_D, T_\xi, N_D, \Lambda,\psi_\xi)$ in (1.4.11),} \\ &\text{$(\mathrm{GSp}_2, T_S, N, \Lambda,\psi_S)$ in (2.3.1) or} \\ &\text{$(\mathrm{GSU}_{3,D}, M_X, N_{4,2},\chi,\psi_{X})$ in (2.3.3).} \end{align*}

Let $(\pi, V_\pi )$ be an irreducible tempered representation of $H=H(k)$ with trivial central character and $[\,,\,]$ a $H$-invariant hermitian pairing on $V_\pi$, the space of $\pi$. Let us denote by $V_\pi ^\infty$ the space of smooth vectors in $V_\pi$. When $k$ is non-archimedean, clearly $V_\pi ^\infty = V_\pi$. Let $\chi$ be a character of $T=T(k)$ which is trivial on $Z_H=Z_H(k)$, where $Z_H$ denotes the center of $H$.

Suppose that $k$ is non-archimedean. Then for $\phi,\phi ^\prime \in V_\pi$, we define the local Bessel period $\alpha _{\chi, \psi _N}^H(\phi, \phi ^\prime ) = \alpha _{\chi, \psi _N}(\phi, \phi ^\prime ) = \alpha (\phi,\phi ^\prime )$ by

(2.4.1)\begin{equation} \alpha(\phi,\phi^\prime):=\int_{T / Z_H }\int_{N}^{\mathrm{st}} [\pi (ut)\phi,\phi^\prime]\,\chi(t)^{-1} \psi_N(u)^{-1}\, du\,dt. \end{equation}

Here the inner integral of (2.4.1) is the stable integral in the sense of Lapid and Mao [LM15, Definition 2.1, Remark 2.2]. Indeed it is shown that for any $t\in T$ the inner integral stabilizes at a certain compact open subgroup of $N=N(k)$ and the outer integral converges by Liu [Liu16, Proposition 3.1, Theorem 2.1]. We note that it is also shown in Waldspurger [Wal12a, § 5.1, Lemme] that (2.4.1) is well-defined. We often simply write $\alpha (\phi ) = \alpha (\phi, \phi )$.

Now suppose that $k$ is archimedean. Then the local Bessel period is defined as a regularized integral whose regularization is achieved by the Fourier transform as in Liu [Liu16, 3.4]. Let us briefly recall the definition. We define a subgroup $N_{-\infty }$ of $N=N(k)$ by

\begin{align*} N_{-\infty}&:= \bigg\{\!\begin{pmatrix}1 & u\\0 & 1\end{pmatrix}\in N_D: \mathrm{tr}_D(\xi u)=0\bigg\}\quad\text{in the $G_D$-case}, \\ N_{-\infty}&:= \bigg\{\!\begin{pmatrix}1 & Y\\0 & 1\end{pmatrix}\in N: \mathrm{tr}(SY)=0\bigg\}\quad\text{in the $\mathrm{GSp}_2$-case},\\ N_{-\infty}&:= \left\{\!\begin{pmatrix}1 & A^\prime & B\\0 & 1 & A\\0 & 0 & 1\end{pmatrix}\in N_{3,D}: \mathrm{tr}_D(XA)=0\right\}\quad\text{in the $\mathrm{GSU}_{3,D}$-case}, \end{align*}

respectively. Then it is shown in Liu [Liu16, Corollary 3.13] that for $u \in N$,

\[ \alpha_{\phi, \phi^\prime}(u):=\int_{T / Z_G} \int_{N_{-\infty}} [\pi(us t)\phi,\phi^\prime]\,\chi(t)^{-1} \, ds \, dt \]

converges absolutely for $\varphi,\varphi ^\prime \in V_\pi ^\infty$ and it gives a tempered distribution on $N / N_{-\infty }$.

For an abelian Lie group $\mathcal {N}$, we denote by $\mathcal {D}(\mathcal {N})$ (respectively, $\mathcal {S}(\mathcal {N})$) the space of tempered distributions (respectively, Schwartz functions) on $\mathcal {N}$. Then we recall that the Fourier transform $\hat {}: \mathcal {D}(\mathcal {N}) \rightarrow \mathcal {D}(\mathcal {N})$ is defined by the formula

\[ (\hat{\mathfrak{a}}, \phi) = (\mathfrak{a}, \hat{\phi})\quad\text{for $\mathfrak a\in \mathcal{D}(\mathcal{N})$ and $\phi\in \mathcal{S}(\mathcal{N})$}, \]

where $(\,{,}\,)$ denotes the natural pairing $\mathcal {D}(\mathcal {N}) \times \mathcal {S}(\mathcal {N}) \rightarrow \mathbb {C}$ and $\hat {\phi }$ is the Fourier transform of $\phi \in \mathcal {S}(\mathcal {N})$.

Then by Liu [Liu16, Proposition 3.14], the Fourier transform $\widehat {\alpha _{\phi, \phi ^\prime }}$ is smooth on the regular locus $(\widehat {N / N_{-\infty }})^{\rm reg}$ of the Pontryagin dual $\widehat {N / N_{-\infty }}$ and we define the local Bessel period $\alpha (\phi,\phi ^\prime )$ by

(2.4.2)\begin{equation} \alpha_{\chi, \psi_N}^H(\phi, \phi^\prime) = \alpha_{\chi, \psi_N}(\phi, \phi^\prime) = \alpha(\phi,\phi^\prime) := \widehat{\alpha_{\phi, \phi^\prime}} (\psi_N). \end{equation}

As in the non-archimedean case, we often simply write $\alpha (\phi ) = \alpha (\phi, \phi )$.

3. Pull-back of Bessel periods

In this section, we establish the pull-back formulas of the global Bessel periods with respect to the dual pairs, $(\mathrm {GSp}_2,\mathrm {GSO}_{4,2})$, $(\mathrm {GSp}_2,\mathrm {GSO}_{3,3})$ and $(G_D,\mathrm {GSU}_{3,D})$. We recall that the first two cases may be regarded as the special case when $D$ is split of the last one, by the accidental isomorphisms explained in § 2.2.

3.1 $(\mathrm {GSp}_2,\mathrm {GSO}_{4,2})$ and $(\mathrm {GSp}_2,\mathrm {GSO}_{3,3})$ case

3.1.1 Symplectic-orthogonal theta correspondence with similitudes

Let $X$ (respectively, $Y$) be a finite-dimensional vector space over $F$ equipped with a non-degenerate alternating (respectively, symmetric) bilinear form. Assume that $\dim _F Y$ is even. We denote their similitude groups by $\mathrm {GSp}(X)$ and $\mathrm {GO}(Y)$, and, their isometry groups by $\mathrm {Sp}(X)$ and $\mathrm {O}(Y)$, respectively. We denote the identity component of $\mathrm {GO}(Y)$ and $\mathrm {O}(Y)$ by $\mathrm {GSO}(Y)$ and $\mathrm {SO}(Y)$, respectively. We let $\mathrm {GSp}(X)$ (respectively, $\mathrm {GO}(Y)$) act on $X$ from right (respectively, left). The space $Z = X \otimes Y$ has a natural non-degenerate alternating form $\langle \,{,} \, \rangle$, and we have an embedding $\mathrm {Sp}(X) \times \mathrm {O}(Y) \rightarrow \mathrm {Sp}(Z)$ defined by

(3.1.1)\begin{equation} (x \otimes y)(g, h) = x g \otimes h^{-1}y,\quad \text{for } x \in X,\enspace y \in Y,\enspace h \in \mathrm{O}(Y),\enspace g \in \mathrm{Sp}(X). \end{equation}

Fix a polarization $Z = Z_{+} \oplus Z_{-}$. Let us denote by $(\omega _{\psi }, \mathcal {S}(Z_{+}(\mathbb {A})))$ the Schrödinger model of the Weil representation of $\widetilde {\mathrm {Sp}}(Z)$ corresponding to this polarization with the Schwartz–Bruhat space $\mathcal {S}(Z_{+})$ on $Z_{+}$. We denote a typical element of $\mathrm {Sp}(Z)$ by

\[ \begin{pmatrix} A & B\\ C & D\\ \end{pmatrix},\quad\text{where}\ \begin{cases} A \in {\rm Hom}(Z_{+} , Z_{+}), & B \in {\rm Hom}(Z_{+} , Z_{-}),\\ C \in {\rm Hom}(Z_{-} , Z_{+}), & D \in {\rm Hom}(Z_{-} , Z_{-}). \end{cases} \]

Then the action of $\omega _{\psi }$ on $\phi \in \mathcal {S}(Z_+)$ is given by the following formulas:

(3.1.2)\begin{equation} \omega_{\psi}\bigg(\begin{pmatrix} A & B\\ 0 & ^{t}A^{-1}\\ \end{pmatrix},\varepsilon\bigg)\phi(z_{+})= \varepsilon \frac{\gamma _{\psi }(1)}{\gamma _{\psi }({\rm det}A)} |{\rm det}(A)|^{1/2} \psi \bigg(\frac{1}{2} \langle z_{+}A ,z_{+}B \rangle\bigg)\phi(z_{+}A), \end{equation}

(3.1.3)\begin{equation} \omega_{\psi}\bigg(\begin{pmatrix} 0 & I\\ -I & 0 \end{pmatrix},\varepsilon\bigg)\phi(z_{+}) =\varepsilon (\gamma _{\psi }(1))^{-\dim Z_{+}} \int _{Z_{+}} \psi\bigg(\bigg\langle z^{\prime}, z \begin{pmatrix} 0 & I\\ -I & 0\\ \end{pmatrix}\bigg\rangle \bigg)\phi(z^{\prime}) \, d z^{\prime}, \end{equation}

where $\gamma _{\psi }(t)$ is a certain eighth root of unity called the Weil factor. Moreover, since the embedding given by (3.1.1) splits in the metaplectic group $\mathrm {Mp}(Z)$, we obtain the Weil representation of $\mathrm {Sp}(X, \mathbb {A}) \times \mathrm {O}(Y, \mathbb {A})$ by restriction. We also denote this representation by $\omega _\psi$.

We have a natural homomorphism

\[ i : \mathrm{GSp}(X) \times \mathrm{GO}(Y) \rightarrow \mathrm{GSp}(Z) \]

given by the action (3.1.1). Then we note that $\lambda (i(g,h)) = \lambda (g)\lambda (h)^{-1}$. Let

\[ R:=\{(g,h) \in \mathrm{GSp}(X)\times \mathrm{GO}(Y) \,|\, \lambda(g) = \lambda(h)\}\supset \mathrm{Sp}(X) \times \mathrm{O}(Y). \]

We may define an extension of the Weil representation of $\mathrm {Sp}(X, \mathbb {A}) \times \mathrm {O}(Y, \mathbb {A})$ to $R(\mathbb {A})$ as follows. Let $X= X_+ \oplus X_-$ be a polarization of $X$ and use the polarization $Z_{\pm } = X_{\pm } \otimes Y$ of $Z$ to realize the Weil representation $\omega _\psi$. Then we note that

\[ \omega_{\psi}(1, h)\phi(z) = \phi(i (h)^{-1} z)\quad \text{for $h\in\mathrm{O}(\mathbb{A})$ and $\phi\in \mathcal{S}(Z_+(\mathbb{A}))$}. \]

Thus, we define an action $L$ of $\mathrm {GO}(Y,\mathbb {A})$ on $\mathcal {S}(Z_+(\mathbb {A}))$ by

\[ L(h)\phi(z)=|\lambda(h)|^{-({1}/{8}) \dim~X \cdot \dim~Y}\phi(i (h)^{-1}z). \]

Then we may extend the Weil representation $\omega _\psi$ of $\mathrm {Sp}(X, \mathbb {A}) \times \mathrm {O}(Y, \mathbb {A})$ to $R(\mathbb {A})$ by

\[ \omega_{\psi}(g, h) \phi = \omega_\psi(g_1, 1) L(h)\phi\quad \text{for $\phi\in\mathcal{S}(Z_+(\mathbb{A}))$ and $(g,h)\in R(\mathbb{A})$}, \]

where

\[ g_1 = g \begin{pmatrix}\lambda(g)^{-1} & 0\\ 0 & 1 \end{pmatrix} \in \mathrm{Sp}(X,\mathbb{A}). \]

In general, for any polarization $Z= Z_+^\prime \oplus Z_-^\prime$, there exists an $\mathrm {Sp}(X, \mathbb {A}) \times \mathrm {O}(Y)(\mathbb {A})$-isomorphism $p : \mathcal {S}(Z_+(\mathbb {A})) \to \mathcal {S}(Z_+^\prime (\mathbb {A}))$ given by an integral transform (see Ichino and Prasanna [IP21, Lemma 3.3]). Let us denote the realization of the Weil representation of ${\mathrm {Sp}(X, \mathbb {A}) \times \mathrm {O}(Y)(\mathbb {A})}$ on $\mathcal {S}(Z_+^\prime (\mathbb {A}))$ by $\omega _\psi ^\prime$. Then we may extend $\omega _\psi ^\prime$ to $R(\mathbb {A})$ by

\[ \omega_\psi^\prime(g,h)=p\circ \omega_\psi(g,h)\circ p^{-1}\quad \text{for $(g, h) \in R(\mathbb{A})$}. \]

For $\phi \in \mathcal {S}(Z_{+}(\mathbb {A}))$, we define the theta kernel $\theta ^\phi$ by

\[ \theta_{\psi}^{\phi}(g , h) = \theta^{\phi}(g , h) := \sum_{z_{+} \in Z_{+}(F)} \omega_{\psi} (g,h)\phi(z_{+}) \quad \text{for $(g, h) \in R(\mathbb{A})$}. \]

Let

(3.1.4)\begin{equation} \mathrm{GSp}(X, \mathbb{A})^{+}= \{g \in \mathrm{GSp}(X, \mathbb{A}) \mid \, \lambda(g) = \lambda(h) \text{ for some } h \in \mathrm{GO}(Y, \mathbb{A})\} \end{equation}

and $\mathrm {GSp}(X, F)^{+} = \mathrm {GSp}(X, \mathbb {A})^{+} \cap \mathrm {GSp}(X, F)$.

As in [HK92, § 5.1], for a cusp form $f$ on $\mathrm {GSp}(X, \mathbb {A})^{+}$, we define its theta lift to $\mathrm {GO}(Y, \mathbb {A})$ by

\[ \Theta_\psi^{X, Y} (f, \phi)(h) = \Theta(f, \phi)(h) := \int_{\mathrm{Sp}(X, F) \backslash \mathrm{Sp}(X, \mathbb{A})} \theta ^{\phi}(g_1g , h)f(g_1 g) \, dg_1 \]

for $h \in \mathrm {GO}(Y, \mathbb {A})$, where $g \in \mathrm {GSp}(X, \mathbb {A})^+$ is chosen so that $\lambda (g) = \lambda (h)$. It defines an automorphic form on $\mathrm {GO}(Y, \mathbb {A})$. For a cuspidal automorphic representation $(\pi _+, V_{\pi _+})$ of $\mathrm {GSp}(X, \mathbb {A})^+$, we denote by $\Theta _\psi (\pi _+)$ the theta lift of $\pi _+$ to $\mathrm {GO}(Y, \mathbb {A})$. Namely,

\[ \Theta_\psi^{X, Y}(\pi_+) = \Theta_\psi(\pi_+): = \{\Theta (f, \phi) : f \in V_{\pi_+}, \phi \in \mathcal{S}(Z_+(\mathbb{A}))\}. \]

Furthermore, for an irreducible cuspidal automorphic representation $(\pi, V_\pi )$ of $\mathrm {GSp}(X, \mathbb {A})$, we define

\[ \Theta_\psi(\pi): = \Theta_\psi(\pi|_{\mathrm{GSp}(X, \mathbb{A})^+}), \]

where $\pi |_{\mathrm {GSp}(X, \mathbb {A})^+}$ denotes the automorphic representation of $\mathrm {GSp}(X, \mathbb {A})^+$ with its space of automorphic forms $\{\varphi |_{\mathrm {GSp}(X, \mathbb {A})^+} : \varphi \in V_\pi \}$.

As for the opposite direction, for a cusp form $f^\prime$ on $\mathrm {GO}(Y, \mathbb {A})$, we define its theta lift $\Theta (f^\prime, \phi )$ to $\mathrm {GSp}(X, \mathbb {A})^+$ by

\[ \Theta (f^\prime, \phi)(g): = \int _{\mathrm{O}(Y, F) \backslash \mathrm{O}(Y, \mathbb{A})} \theta ^{\phi}(g , h_1h)f(h_1 h) \, dh_1\quad \text{for $g\in\mathrm{GSp}(X,\mathbb{A})^+$}, \]

where $h \in \mathrm {GO}(Y,\mathbb {A})$ is chosen so that $\lambda (g) = \lambda (h)$. For an irreducible cuspidal automorphic representation $(\sigma, V_\sigma )$ of $\mathrm {GO}(Y, \mathbb {A})$, we define the theta lift $\Theta _\psi (\sigma )$ of $\sigma$ to $\mathrm {GSp}(X, \mathbb {A})^+$ by

\[ \Theta_\psi(\sigma):=\{\Theta (f^\prime, \phi): f^\prime\in V_\sigma,\ \phi \in \mathcal{S}(Z_+(\mathbb{A}))\}. \]

Moreover, we extend $\theta (f^\prime, \phi )$ to an automorphic form on $\mathrm {GSp}(X, \mathbb {A})$ by the natural embedding

\[ \mathrm{GSp}(X, F)^+ \backslash \mathrm{GSp}(X, \mathbb{A})^+ \rightarrow \mathrm{GSp}(X, F) \backslash \mathrm{GSp}(X, \mathbb{A}) \]

and extension by zero. Then we define the theta lift $\Theta _\psi (\sigma )$ of $\sigma$ to $\mathrm {GSp}(X, \mathbb {A})$ as the $\mathrm {GSp}(X,\mathbb {A})$ representation generated by such $\theta (f^\prime,\phi )$ for $f^\prime \in V_\sigma$ and $\phi \in \mathcal {S}(Z_+(\mathbb {A}))$.

For some $X$ and $Y$, theta correspondence for the dual pair $(\mathrm {GSp}(X)^+, \mathrm {GO}(Y))$ gives theta correspondence between $\mathrm {GSp}(X)^+$ and $\mathrm {GSO}(Y)$ by the restriction of representations of $\mathrm {GO}(Y)$ to $\mathrm {GSO}(Y)$. Indeed, when $\dim X=4$ and $\dim Y = 6$, we may consider theta correspondence for the pair $(\mathrm {GSp}(X)^+, \mathrm {GSO}(Y))$. In Gan and Takeda [GT10, GT11b], they study the case when $\mathrm {GSO}(Y) \simeq \mathrm {GSO_{3,3}}$ or $\mathrm {GSO}_{5,1}$, and, in [Mor14a], the case when $\mathrm {GSO}(Y) \simeq \mathrm {GSO_{4,2}}$ is studied. In these cases, for a cusp form $f$ on $\mathrm {GSp}(X, \mathbb {A})^{+}$, we denote by $\theta (f, \phi )$ the restriction of $\Theta (f, \phi )$ to $\mathrm {GSO}(Y, \mathbb {A})$. Moreover, for a cuspidal automorphic representation $(\pi _+, V_{\pi _+})$ of $\mathrm {GSp}(X, \mathbb {A})^+$, we define the theta lift $\theta _\psi (\pi _+)$ of $\pi _+$ to $\mathrm {GSO}(Y, \mathbb {A})$ by

\[ \theta_\psi^{X, Y}(\pi_+) = \theta_\psi(\pi_+) := \{\theta (f, \phi) : f \in V_{\pi_+},\, \phi \in \mathcal{S}(Z_+(\mathbb{A}))\}. \]

Similarly, for a cusp form $f^\prime$ on $\mathrm {GSO}(Y, \mathbb {A})$, we define its theta lift $\theta (f^\prime, \phi )$ to $\mathrm {GSp}(X, \mathbb {A})^+$ by

\[ \theta (f^\prime, \phi)(g): = \int _{\mathrm{SO}(Y, F) \backslash \mathrm{SO}(Y, \mathbb{A})} \theta^{\phi}(g , h_1h)f(h_1 h) \, dh_1\quad \text{for $g\in\mathrm{GSp}(X,\mathbb{A})^+$}, \]

where $h \in \mathrm {GSO}(Y,\mathbb {A})$ is chosen so that $\lambda (g) = \lambda (h)$. We extend it to an automorphic from on $\mathrm {GSp}(X, \mathbb {A})$ as above. For a cuspidal automorphic representation $(\sigma, V_\sigma )$ of $\mathrm {GSO}(Y, \mathbb {A})$, we define the theta lift $\theta _\psi (\sigma )$ of $\sigma$ to $\mathrm {GSp}(X, \mathbb {A})^+$ by

\[ \theta_\psi(\sigma):= \{\theta (f^\prime, \phi): f^\prime\in V_\sigma,\ \phi \in \mathcal{S}(Z_+(\mathbb{A}))\}. \]

Remark 3.1 Suppose that $\Theta _\psi (\pi _+)$ (respectively, $\theta _\psi (\sigma )$) is non-zero and cuspidal where $(\pi _+, V_{\pi _+})$ (respectively, $(\sigma, V_\sigma )$) is an irreducible cuspidal automorphic representation of $\mathrm {GSp}(X, \mathbb {A})^+$ (respectively, $\mathrm {GO}(Y, \mathbb {A})$). Then Gan [Gan08, Proposition 2.12] has shown that the Howe duality, which was proved by Howe [How89] at archimedean places, by Waldspurger [Wal90] at odd finite places and finally by Gan and Takeda [GT16] at all finite places, implies that $\Theta _\psi (\pi _+)$ (respectively, $\theta _\psi (\sigma )$) is irreducible and cuspidal. Moreover, in the case of our concern, namely when $\dim _FX=4$ and $\dim _FY=6$, the irreducibility of $\Theta _\psi (\pi _+)$ implies that of $\theta _\psi (\pi _+)$ by the conservation relation due to Sun and Zhu [SZ15].

3.1.2 Pull-back of the global Bessel periods for the dual pairs $(\mathrm {GSp}_2,\mathrm {GSO}_{4,2})$ and $(\mathrm {GSp}_2,\mathrm {GSO}_{3,3})$

Our goal here is to prove the pull-back formula (3.1.6).

First we introduce the set-up. Let $X$ be the space of four-dimensional row vectors over $F$ equipped with the symplectic form

\[ \langle w_1,w_2\rangle=w_1\begin{pmatrix}0 & 1_2\\-1_2 & 0\end{pmatrix}\,{}^tw_2. \]

Let us take the standard basis of $X$ and name the basis vectors as

(3.1.5)\begin{equation} x_1 = (1, 0, 0, 0),\quad x_2 = (0, 1, 0, 0),\quad x_{-1} = (0, 0, 1, 0),\quad x_{-2} = (0, 0, 0, 1). \end{equation}

Then the matrix representation of $\mathrm {GSp}(X)$ with respect to the standard basis is $G=\mathrm {GSp}_2$ defined by (1.4.3). We let $G$ act on $X$ from the right.

Let $Y$ be the space of six-dimensional column vectors over $F$ equipped with the non-degenerate symmetric bilinear form

\[ (v_1,v_2)={}^tv_1 S_2 v_2, \]

where the symmetric matrix $S_2$ is given by (2.1.2). Let us take the standard basis of $Y$ and name the basis vectors as

\begin{align*} y_{-2} &= {}^{t}(1, 0, 0, 0, 0, 0),\quad y_{-1} = {}^{t}(0, 1, 0, 0, 0, 0),\\ e_1&= {}^{t}(0, 0, 1, 0, 0, 0),\quad e_2 = {}^{t}(0, 0, 0, 1, 0, 0),\\ y_{1} &= {}^{t}(0, 0, 0, 0, 1, 0),\quad y_{2} = {}^{t}(0, 0, 0, 0, 0, 1). \end{align*}

We note that $( y_i , y_j )=\delta _{ij}, ( e_1 , e_1)=2$ and $( e_2 , e_2) =-2d$. Since $d \in F^\times \setminus (F^\times )^2$, with respect to the standard basis, the matrix representations of $\mathrm {GO}(Y)$ and $\mathrm {GSO}(Y)$ are $\mathrm {GO}_{4,2}$ defined by (2.1.5) and $\mathrm {GSO}_{4,2}$ defined by (2.1.4), respectively. In this section, we also study the theta correspondence for the dual pair $(\mathrm {GSp}(X), \mathrm {GSO}_{3,3})$, for which, we may use the above matrix representation with $d \in (F^\times )^2$. Hence, in the remaining of this section, we study theta correspondence for $(\mathrm {GSp}(X), \mathrm {GSO}(Y))$ for an arbitrary $d \in F^\times$.

We shall denote $\mathrm {GSp}(X, \mathbb {A})^+$ by $G(\mathbb {A})^+$ and also $\mathrm {GSp}(X, F)^+$ by $G(F)^+$. We note that when $d \in (F^\times )^2$, $\mathrm {GSp}(X)^+ = \mathrm {GSp}(X)$.

Let $Z=X\otimes Y$ and we take a polarization $Z=Z_+\oplus Z_-$ as follows. First we take $X=X_+\oplus X_-$ where

\[ X_{+} = F \cdot x_1 + F \cdot x_2 \quad \text{and} \quad X_{-}= F \cdot x_{-1} + F \cdot x_{-2} \]

as the polarization of $X$. Then we decompose $Y$ as $Y=Y_+\oplus Y_0\oplus Y_-$ where

\[ Y_{+} =F \cdot y_1+ F \cdot y_2,\quad Y_0 = F \cdot e_1 + F \cdot e_2\quad \text{and}\quad Y_{-} =F \cdot y_{-1} + F \cdot y_{-2}. \]

Then let

\[ Z_{\pm} = (X \otimes Y_{\pm})\oplus(X_{\pm} \otimes Y_0), \]

where the double sign corresponds. To simplify the notation, we sometimes write $z_+\in Z_+$ as $z_+=(a_1,a_2;b_1,b_2)$ when

\[ z_{+} = a_1 \otimes y_1 + a_2 \otimes y_2 + b_1 \otimes e_1 + b_2 \otimes e_2\in Z_+,\quad \text{where $ a_i \in X,\ b_i \in X_{+}\, (i=1,2)$}. \]

Let us compute the pull-back of $(X, \chi, \psi )$-Bessel periods on $\mathrm {GSO}(Y)$ defined by (2.3.6) with respect to the theta lift from $G$.

Proposition 3.1 Let $(\pi,V_\pi )$ be an irreducible cuspidal automorphic representation of $G(\mathbb {A})$ whose central character is $\omega _\pi$ and $\chi$ a character of $\mathbb {A}_E^\times$ such that $\chi \mid _{\mathbb {A}^\times }=\omega _\pi ^{-1}$. Let $X\in \mathrm {Mat}_{2\times 2}(F)$ such that $\det X\ne 0$.

Then for $f \in V_\pi$ and $\phi \in \mathcal {S}(Z_+(\mathbb {A}))$, we have

(3.1.6)\begin{equation} \mathcal{B}_{X, \chi, \psi}(\theta(f: \phi)) = \int_{N(\mathbb{A}) \backslash G^1(\mathbb{A})} B_{S_X, \chi^{-1},\psi}(\pi(g)f)(\omega_\psi(g, 1) \phi)(v_X) \, dg, \end{equation}

where $B_{S_X, \chi ^{-1},\psi }$ is the $(S_X,\chi ^{-1},\psi )$-Bessel period on $G$ defined by (2.3.1).

Here, for $X= \big (\begin{smallmatrix}x_{11} & x_{12}\\x_{21} & x_{22} \end{smallmatrix}\big )$, we define a vector $v_X\in Z_+$ by

(3.1.7)\begin{equation} v_X := \bigg(x_{-2}, x_{-1};\frac{x_{21}}{2}x_1+\frac{x_{11}}{2} x_2, -\frac{x_{22}}{2d}x_1-\frac{x_{12}}{2d} x_2\bigg) \end{equation}

and a $2$-by-$2$ symmetric matrix $S_X$ by

(3.1.8)\begin{equation} S_X: = \frac{1}{4d}{}^{t}(J_2 \,{}^{t}X J_2)S_0 ( J_2\,{}^{t}X J_2). \end{equation}

We regard $\chi$ as a character of $\mathrm {GSO}(S_X)(\mathbb {A})$ by

(3.1.9)\begin{equation} \mathrm{GSO}(S_X) \ni k \mapsto \chi((J_2{}^{t}XJ_2) k (J_2{}^{t}XJ_2)^{-1})\in\mathbb{C}^\times. \end{equation}

In particular, the $(S_X, \chi ^{-1},\psi )$-Bessel period does not vanish on $V_\pi$ if and only if the $(X,\chi, \psi )$-Bessel period does not vanish on $\theta _\psi (\pi )$.

Proof. We compute the $(X,\chi, \psi )$-Bessel period defined by (2.3.6) in stages. We consider subgroups of $N_{4,2}$ given by

(3.1.10)\begin{align} N_0(F) &= \left\{u_0(x) :=\left.\begin{pmatrix} 1 & -{}^{t}X_0 S_1 & 0\\ 0 & 1_4 & X_0\\ 0 & 0 & 1 \end{pmatrix}\right| X_0 = \begin{pmatrix} x\\0\\0\\0\end{pmatrix}\right\}; \end{align}

(3.1.11)\begin{align} N_1(F) &= \left\{u_1(s_1, t_1) :=\left.\begin{pmatrix} 1 & -{}^{t}X_1 S_1 & -\dfrac{1}{2}{}^{t}X_1 S_1 X_1\\ 0 & 1_4 & X_1\\ 0 & 0 & 1 \end{pmatrix}\right| X_1 = \begin{pmatrix} 0\\s_1\\t_1\\0 \end{pmatrix}\right\}; \end{align}

(3.1.12)\begin{align} N_2(F) &= \left\{u_2(s_2, t_2) := \left.\begin{pmatrix} 1 & 0 & 0 & 0 & 0\\ 0 & 1 & -{}^{t}X_2 S_0 & -\dfrac{1}{2}{}^{t}X_2 S_0 X_2 & 0\\ 0 & 0 & 1_2 & X_2 & 0\\ 0 & 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 0 & 1 \end{pmatrix}\right| X_2 = \begin{pmatrix} s_2\\ t_2 \end{pmatrix}\right\}, \end{align}

where $S_0$ and $S_1$ are given by (2.1.2). Then we have

\[ N_0 \lhd N_0 N_1 \lhd N_0 N_1 N_2 =N_{4.2}. \]

Thus, we may write

(3.1.13)\begin{align} \mathcal{B}_{X, \chi, \psi}(\theta(f: \phi)) &= \int_{\mathbb{A}^\times M_{X}(F) \backslash M_X(\mathbb{A})} \int_{(F \backslash \mathbb{A}_{F})^2} \int_{(F \backslash \mathbb{A}_{F})^2} \int_{F \backslash \mathbb{A}_{F}} \theta(f, \phi)(u_0(x) u_1(s_1, t_1) u_2(s_2, t_2) h) \nonumber\\ &\quad \times \psi(x_{21}s_1+x_{22}t_1+x_{11}s_2+x_{12}t_2)^{-1}\chi (h)^{-1} \, {dx} \,ds_1 \,dt_1 \,ds_2 \,dt_2 \,dh. \end{align}

For $h \in \mathrm {GSO}(Y, \mathbb {A})$, let us define

\[ W_0(\theta(f: \phi))(h):= \int_{F \backslash \mathbb{A}_F} \theta(f, \phi)(u_0(x)h)\,{dx}. \]

From the definition of the theta lift, we have

(3.1.14)\begin{align} & W_0(\theta(f, \phi))(h) \nonumber\\ &\quad =\int_{F \backslash \mathbb{A}_F} \int_{G^1(F) \backslash G^1(\mathbb{A}_F)} \sum_{a_i \in X, b_i \in X_{+}} (\omega_\psi(g_1 \lambda_s(\nu(h)), u_0(x) h) \phi)(a_1, a_2; b_1, b_2) \nonumber\\ &\qquad \times f(g_1 \lambda_s(\lambda(h)))\, dg_1 \, {dx}. \end{align}

Here, for $a \in \mathbb {A}^\times$, we let

\[ \lambda_s(a) = \begin{pmatrix} 1_2 & 0\\0 & a \cdot 1_2 \end{pmatrix}. \]

Since $Z_{-}(1, u_0(x))= Z_{-}$ and we have

\[ z_{+}(1, u_0(x)) = z_{+} + (x \cdot a_1 \otimes y_{-2} -x \cdot a_2 \otimes y_{-1}), \]

we observe that

(3.1.15)\begin{align} (\omega_\psi(1, u_0(x)) \phi)(z_+) &= \psi \big(\tfrac{1}{2} \langle z_+, x \cdot a_1 \otimes y_{-2} -x \cdot a_2 \otimes y_{-1} \rangle \big) \phi(z_+) \nonumber\\ &= \psi (-x \langle a_1, a_2 \rangle) \phi(z_+). \end{align}

Thus, in the summation of the right-hand side of (3.1.14), only $a_i$ such that $\langle a_1, a_2 \rangle = 0$ contributes to the integral $W_0(\theta (f, \phi ))$, and we obtain

\begin{align*} W_0(\theta(f, \phi))(h) &= \int_{G^1(F) \backslash G^1(\mathbb{A}_F)}\\ &\quad \times\sum_{\substack{a_i \in X, \langle a_1, a_2 \rangle =0,\\ b_i \in X_{+}}} (\omega_\psi(g_1 \lambda_s(\lambda(h)), h) \phi) (a_1, a_2; b_1, b_2) f(g_1 \lambda_s(\lambda(h)))\,dg_1. \end{align*}

Since the space spanned by $a_1$ and $a_2$ is isotropic, there exists $\gamma \in G^1(F)$ such that $a_1 \gamma ^{-1}, a_2 \gamma ^{-1} \in X_{-}$. Let us define an equivalence relation $\sim$ on $(X_{-})^{2}$ by

\[ (a_1 , a_2) \sim (a_1^\prime , a_2^\prime) \underset{\text{def.}}{\Longleftrightarrow} \text{there exists $\gamma\in G^1(F)$ such that $a_i^\prime = a_i \gamma$ for $i=1,2$}. \]

Let us denote by $\mathcal {X}_{-}$ the set of equivalence classes $(X_{-})^2 / \sim$ and by $\overline {(a_1, a_2)}$ the equivalence class containing $(a_1,a_2)\in (X_{-})^2$. Then we may write $W_0 (\theta (f, \phi )) (h)$ as

\begin{align*} & \int_{G^1(F) \backslash G^1(\mathbb{A}_F)} \sum_{\overline{(a_1 , a_2)} \in \mathcal{X}_{-}}\ \sum_{\gamma \in V(a_1 , a_2) \backslash G^1(F)}\ \sum_{b_i \in X_{+}} (\omega_\psi (g_1 \lambda_s(\lambda(h)) , h) \phi)(a_1\gamma, a_2 \gamma; b_1, b_2)\\ &\quad \times f(g_1 \lambda_s(\lambda(h))) \, dg_1. \end{align*}

Here

\[ V(a_1, a_2) = \{g \in G^1(F) \mid \text{$a_ig =a_i$ for $i=1,2$}\}. \]

Lemma 3.1 For any $g \in G(\mathbb {A})^+$ and $h \in \mathrm {GSO}(Y, \mathbb {A})$ such that $\lambda (g) = \lambda (h)$,

\[ \sum_{b_i \in X_{+}}(\omega_\psi (g, h) \phi)(a_1 \gamma, a_2 \gamma, b_1, b_2) =\sum_{b_i \in X_{+}}(\omega_\psi (\gamma g, h) \phi)(a_1 , a_2 , b_1, b_2). \]

Proof. This is proved by an argument similar to that for [Fur95, Lemma 2].

Further, by an argument similar to the one for $W_0 (\theta (f, \phi )) (h)$, we shall prove the following lemma.

Lemma 3.2 For any $g \in G(\mathbb {A})^+$ and $h \in \mathrm {GSO}(Y, \mathbb {A})$ such that $\lambda (g) = \lambda (h)$,

\begin{align*} & \int_{(F \backslash \mathbb{A}_F)^{2}} \psi^{-1}(x_{21}s_1+x_{22}t_1) (\omega_\psi (g , u_{1}(s_1, t_1) h) \phi)(a_1 , a_2 , b_1, b_2)\,ds_1\,dt_1\\ &\quad = \left\{\begin{array}{@{}ll} (\omega_\psi (g , h)\phi)(a_1 , a_2 , b_1, b_2) & \text{if $ \langle a_2, b_1 \rangle =-\dfrac{x_{21}}{2}$ and $\langle a_2, b_2 \rangle= \dfrac{x_{22}}{2d}$};\\ & \\ 0 & \text{otherwise}\\ \end{array}\right. \end{align*}

and

\begin{align*} & \int_{(F \backslash \mathbb{A}_F)^{2}} \psi^{-1}(x_{11} s_2+x_{12}t_2) (\omega_\psi(g , u_{2}(s_2, t_2) h) \phi)(a_1 , a_2 , b_1, b_2)\, ds_2\, dt_2\\ &\quad =\left\{\begin{array}{@{}ll} (\omega_\psi (g , h)\phi)(a_1 , a_2 , b_1, b_2) & \text{if $ \langle a_1 , b_1 \rangle =-\dfrac{x_{11}}{2}$ and $\langle a_1, b_2 \rangle =\dfrac{x_{12}}{2d}$};\\ & \\ 0 & \text{otherwise}.\\ \end{array}\right. \end{align*}

Proof. Since $Z_-\, (1, u_1(s_1, t_1)) = Z_-$ and we have

\begin{align*} z_+(1, u_1(s_1, t_1)) &= z_{+} +2s_1 (b_1 \otimes y_{-2}) - 2dt_1(b_2 \otimes y_{-2})\\ &\quad +(-s_1^2+2dt_1^2)a_2 \otimes y_{-2}-s_1a_2 \otimes e_1-t_1 a_2 \otimes e_2, \end{align*}

we obtain

\begin{align*} & (\omega_\psi(1, u_1(s_1, t_1)) \phi)(z_+) = \psi \big(\tfrac{1}{2} (2s_1\langle a_2, b_1 \rangle -2dt_1 \langle a_2, b_2 \rangle)\big)\\ &\qquad \times\psi \big(\tfrac{1}{2} ((-s_1^2+2dt_1^2) \langle a_2, a_2 \rangle -2s_1 \langle b_1, a_2 \rangle +2dt_1 \langle b_2, a_2 \rangle)\big)\phi(z_+)\\ &\quad =\psi (2s_1\langle a_2, b_1 \rangle -2dt_1 \langle a_2, b_2 \rangle)\phi(z_+). \end{align*}

Then the first assertion readily follows.

Similarly, since $Z_- \,(1, u_2(s_2, t_2)) = Z_-$ and we have

\[ z_+\,(1, u_2(s_2, t_2)) = z_+ + a_1 \otimes ((s_2^2-dt_2^2)y_{-1}-s_2 e_1-t_2e_2 ) + 2s_2 b_1 \otimes y_{-1} -2dt_2 b_2 \otimes y_{-1}, \]

we obtain

\begin{align*} & \omega(1, u_2(s_2, t_2)) \phi(z_+)= \psi \big(\tfrac{1}{2} (-2s_2 \langle b_1, a_1 \rangle +2dt_2 \langle b_2, a_1 \rangle)\big)\\ &\qquad \times\psi \big(\tfrac{1}{2}(2s_2 \langle a_1, b_1 \rangle -2dt_2 \langle a_1, b_2 \rangle)\big)\phi(z_+)\\ &\quad =\psi (2s_2 \langle a_1, b_1 \rangle -2dt_2 \langle a_1, b_2 \rangle) \phi(z_+) \end{align*}

and the second assertion follows.

Lemma 3.2 implies that

\begin{align*} \mathcal{B}_{X, \chi, \psi}(\theta(f: \phi)) &= \int_{\mathbb{A}^\times M_{X}(F) \backslash M_X(\mathbb{A})} \int_{G^1(F) \backslash G^1(\mathbb{A}_F)} \chi (h)^{-1}\\ &\quad \times\sum _{ \overline{(a_1 , a_2)} \in \mathcal{X}_{-}} \sum_{\gamma \in V(a_1 , a_2) \backslash G^1(F)} \sum_{\substack{ b_i \in X_{+}, \langle a_i, b_1 \rangle = {{x_{i1}}/{2}},\\ \langle a_i, b_2 \rangle =- ({x_{i2}}/{2d}) }}\\ &\quad \times(\omega_\psi (\gamma g_1 \lambda_s(\lambda(h)) , h) \phi) (a_1, a_2, b_1, b_2) f(g_1 \lambda_s(\lambda(h))) \, dg_1\,dh. \end{align*}

We note that $a_1$ and $a_2$ are linearly independent from the conditions on $a_i$ and $\det (X) \ne 0$. Since $a_i \in X_-$ and $\dim X_- =2$, we may take $(a_1,a_2)=(x_{-2},x_{-1})$ as a representative. Then we should have

\[ b_1 =\frac{x_{21}}{2}x_1+\frac{x_{11}}{2} x_2,\quad b_2= -\frac{x_{22}}{2d}x_1-\frac{x_{12}}{2d} x_2. \]

Hence, we get

(3.1.16)\begin{align} \mathcal{B}_{X, \chi, \psi}(\theta(f: \phi)) &= \int_{\mathbb{A}^\times M_{X}(F) \backslash M_X(\mathbb{A})} \int_{G^1(F) \backslash G^1(\mathbb{A}_F)}\chi(h)^{-1} \nonumber\\ &\quad \times \sum_{\gamma \in N(F) \backslash G^1(F)} (\omega_\psi (\gamma g_1 \lambda_s(\lambda(h)), h) \phi)(v_X) f(g_1 \lambda_s(\lambda(h))) \, dg_1\,dh \nonumber\\ &= \int_{N(\mathbb{A}) \backslash G^1(\mathbb{A}_F)} \int_{\mathbb{A}^\times M_{X}(F) \backslash M_X(\mathbb{A})} \int_{N(F) \backslash N(\mathbb{A})} \nonumber\\ &\quad \times\chi(h)^{-1}\omega (v g_1 \lambda_s(\lambda(h)), h) \phi(v_X) f(v g_1 \lambda_s(\lambda(h))) \, dv\,dg_1\,dh , \end{align}

where we put $v_X = (x_{-2}, x_{-1}; ( {x_{21}}/{2})x_1+{{x_{11}}/{2}} x_2, -({x_{22}}/{2d})x_1-({x_{12}}/{2d}) x_2)$.

For $u=\big (\begin{smallmatrix}1_2 & A\\0 & 1_2\end{smallmatrix}\big )$ where $A= \big (\begin{smallmatrix}a & b\\ b & c\end{smallmatrix}\big ) \in \mathrm {Sym}^2$, we have

\begin{align*} & \bigg( x_{-2} \otimes y_1 + x_{-1} \otimes y_2+ \bigg(\frac{x_{21}}{2}x_1+\frac{x_{11}}{2} x_2 \bigg) \otimes e_1 + \bigg( -\frac{x_{22}}{2d}x_1-\frac{x_{12}}{2d} x_2 \bigg)\otimes e_2 \bigg)(u,1) \\ &\quad = x_{-2} \otimes y_1 + x_{-1} \otimes y_2+ \bigg(\frac{x_{21}}{2}(x_1 +a x_{-1}+b x_{-2} )+ \frac{x_{11}}{2} (x_2+bx_{-1} +c x_{-2})\bigg) \otimes e_1 \\ &\qquad +\bigg(-\frac{x_{22}}{2d}(x_1 +a x_{-1}+b x_{-2} )- \frac{x_{12}}{2d} (x_2+bx_{-1} +c x_{-2})\bigg)\otimes e_2. \end{align*}

Hence, when we put

\begin{align*} S_X &= \frac{1}{4d}{}^{t}(J_2{}^{t}XJ_2)S_0 (J_2{}^{t}XJ_2)\\ &= \frac{1}{2d} \begin{pmatrix} x_{22}^2-dx_{21}^2 & x_{22}x_{12}-dx_{21}x_{11} \\ x_{22}x_{12}-dx_{21}x_{11} & x_{12}^2-dx_{11}^2 \end{pmatrix}\in \mathrm{Sym}^2(F), \end{align*}

for $u\in N(\mathbb {A})$, we have

\[ (\omega_\psi(ug\lambda_s(\lambda(h)), h) \phi)(v_X) = \psi_{S_X} (u)^{-1} \omega_\psi(g\lambda_s(\lambda(h)), h) \phi(v_X). \]

Therefore, we get

\begin{align*} & \int_{N(\mathbb{A}) \backslash G^1(\mathbb{A}_F)} \int_{\mathbb{A}^\times M_{X}(F) \backslash M_X(\mathbb{A})} \int_{N(F) \backslash N(\mathbb{A})} \chi(h)^{-1} \\ &\qquad \times (\omega_\psi (g_1 \lambda_s(\lambda(h)), h) \phi)(v_X) f(u g_1 \lambda_s(\lambda(h))) \psi_{S_X}(u)^{-1} \, du \, dh\, dg_1 \\ &\quad =\int_{N(\mathbb{A}) \backslash G^1(\mathbb{A}_F)} \int_{\mathbb{A}^\times M_{X}(F) \backslash M_X(\mathbb{A})} \int_{N(F) \backslash N(\mathbb{A})} \chi (h)^{-1} \\ &\qquad \times\omega_\psi (\lambda_s(\lambda(h)) g_1, h) \phi(v_X) |\lambda(h)|^3 f(u \lambda_s(\lambda(h)) g_1)\psi_{S_X}(u)^{-1}\, du \, dh\, dg_1. \end{align*}

By a direct computation, we see that

\[ (\omega_\psi (\lambda_s(\lambda(h)) g_1, h) \phi)(v_X) =|\lambda(h)|^{-3} (\omega_\psi (h_0 \lambda_s(\lambda(h)) g_1, 1) \phi)(v_X) \]

when we let

\[ h = \begin{pmatrix} (\det h) (h^X)^\ast & 0 & 0\\ 0 & h & 0\\ 0 & 0 & h^X \end{pmatrix},\quad h_0 = \begin{pmatrix} ({}^{t}XJ_2)^{-1} {}^{t}h ({}^{t}XJ_2) & 0\\ 0 & (J_2X) h^{-1} (J_2X)^{-1} \end{pmatrix}. \]

For $g \in \mathrm {GSO}(S_0)$, we have ${}^{t}g = wgw$ and we may write

\[ h_0 = \begin{pmatrix} (J_2{}^{t}XJ_2)^{-1} {}^{t}h (J_2{}^{t}XJ_2) & 0\\ 0 & {}^{t}((J_2{}^{t}XJ_2)^{-1} {}^{t}h (J_2{}^{t}XJ_2))^{-1} \end{pmatrix}. \]

Since we have

\[ \mathrm{GSO}(S_X) = (J_2{}^{t}XJ_2)^{-1} \mathrm{GSO}(S_0) (J_2{}^{t}XJ_2), \]

we get

(3.1.17)\begin{align} & \int_{N(\mathbb{A}) \backslash G^1(\mathbb{A}_F)} \int_{\mathbb{A}^\times T_{S_X}(F) \backslash T_{S_X}(\mathbb{A})} \int_{N(F) \backslash N(\mathbb{A})} \chi(h) \nonumber\\ &\qquad \times(\omega_\psi (g_1, 1) \phi)(v_X) f(u h g_1) \psi_{S_X}(u)^{-1} \, du \, dh\, dg_1 \nonumber\\ &\quad =\int_{N(\mathbb{A}) \backslash G^1(\mathbb{A}_F)} B_{S_X, \chi^{-1}}(\pi(g_1)f) (\omega_\psi (g_1, 1) \phi)(v_X) \,dg_1 , \end{align}

where we regard $\chi$ as a character of $\mathrm {GSO}(S_X)(\mathbb {A})$ by (3.1.9).

Finally, the last statement concerning the equivalence of the non-vanishing conditions on the $(S_X,\chi ^{-1},\psi )$-Bessel period and the $(X,\chi )$-Bessel period follows from the pull-back formula (3.1.6) by an argument similar to that in the proof of Proposition 2 in [FM17].

3.2 $(G_D, \mathrm {GSU}_{3,D})$ case

3.2.1 Theta correspondence for quaternionic dual pair with similitudes

Let $D$ be a quaternion division algebra over $F$. Let $X_D$ (respectively, $Y_D$) be a right (respectively, left) $D$-vector space of finite rank equipped with a non-degenerate hermitian bilinear form $(\,{,}\,)_{X_D}$ (respectively, non-degenerate skew-hermitian bilinear form $\langle \, {,} \, \rangle _{Y_D}$). Hence $(\,{,}\,)_{X_D}$ and $\langle \, {,} \,\rangle _{Y_D}$ are $D$-valued $F$-bilinear form on $X_D$ and $Y_D$ satisfying

\begin{gather*} \overline{(x ,x^\prime)_{X_D}} = (x^\prime,x)_{X_D},\quad (xa,x^\prime b)_{X_D} = \bar{a}(x,x^\prime)_{X_D} b, \\ \overline{\langle y,y^\prime \rangle_{Y_D}} = -\langle y^\prime,y\rangle_{Y_D}, \quad \langle ay,y^\prime b \rangle_{Y_D} = a \langle y,y^\prime \rangle_{Y_D} \bar{b}, \end{gather*}

for $x, x^\prime \in X_D$, $y, y^\prime \in Y_D$ and $a,b\in D$. We denote the isometry group of $X_D$ and $Y_D$ by $\mathrm {U}(X_D)$ and $\mathrm {U}(Y_D)$, respectively. Then the space $Z_D = X_D \otimes _D Y_D$ is regarded as a symplectic space over $F$ with the non-degenerate alternating form $\langle \,{,} \, \rangle$ defined by

(3.2.1)\begin{equation} \langle x_1 \otimes y_1, x_2 \otimes y_2 \rangle = \mathrm{tr}_D ((x_1, x_2)_{X_D} \overline{\langle y_1, y_2 \rangle_{Y_D}}) \in F \end{equation}

and we have a homomorphism $\mathrm {U}(X_D) \times \mathrm {U}(Y_D) \rightarrow \mathrm {Sp}(Z_D)$ defined by

(3.2.2)\begin{equation} (x \otimes y)(g, h) = x g \otimes h^{-1}y\quad \text{for $x \in X, y \in Y$, $h \in \mathrm{U}(Y_D)$ and $g \in \mathrm{U}(X_D)$.} \end{equation}

As in the case when $D \simeq \mathrm {Mat}_{2 \times 2}$, this mapping splits in the metaplectic group $\mathrm {Mp}(Z_D)$. Hence we have the Weil representation $\omega _\psi$ of $\mathrm {U}(X_D, \mathbb {A}) \times \mathrm {U}(Y_D, \mathbb {A})$ by restriction.

From now on, we suppose that the rank of $X_D$ is $2k$ and $X_D$ is maximally split, in the sense that its maximal isotropic subspace has rank $k$.

Let us denote by $\mathrm {GU}(X_D)$ (respectively, $\mathrm {GU}(Y_D)$) the similitude unitary group of $X_D$ (respectively, $Y_D$) with the similitude character $\lambda _D$ (respectively, $\nu _D$). In addition, we denote the identity component of $\mathrm {GU}(Y_D)$ by $\mathrm {GSU}(Y_D)$. Then the action (3.2.2) extends to a homomorphism

\[ i_D : \mathrm{GU}(X_D) \times \mathrm{GU}(Y_D) \rightarrow \mathrm{GSp}(Z_D) \]

with the property $\lambda (i_D(g, h)) = \lambda _D(g) \nu _D(h)^{-1}$. Let

\[ R_D:=\{(g,h) \in \mathrm{GU}(X_D) \times \mathrm{GU}(Y_D) \,|\, \lambda_D(g) = \nu_D(h)\} \supset \mathrm{U}(X_D) \times \mathrm{U}(Y_D). \]

Since $X_D$ is maximally split, we have a Witt decomposition $X_D = X_D^+ \oplus X_D^-$ with maximal isotropic subspaces $X_{D}^{\pm }$. Then as in § 3.1.1, we may realize the Weil representation $\omega _\psi$ of $\mathrm {U}(X_D) \times \mathrm {U}(Y_D)$ on $\mathcal {S}((X_D^+ \otimes Y_D)(\mathbb {A}))$. In this realization, for $h \in \mathrm {U}(Y_D)$ and $\phi \in \mathcal {S}((X_D^+ \otimes Y_D)(\mathbb {A}))$, we have

\[ \omega_{\psi}(1, h)\phi(z) = \phi(i_D(h)^{-1} z). \]

Hence, as in § 3.1.1, we may extend $\omega _\psi$ to $R_D(\mathbb {A})$ by

\[ \omega_{\psi}(g, h) \phi(z) = |\lambda(h)|^{-2 \mathrm{rank}~X_D \cdot \mathrm{rank}~Y_D} \omega_\psi(g_1, 1) \phi (i_D(h)^{-1}z) \]

for $(g,h)\in R_D(\mathbb {A})$, where

\[ g_1 = g \begin{pmatrix} \lambda_D(g)^{-1} & 0\\ 0 & 1 \end{pmatrix} \in \mathrm{U}(X_D). \]

Then as in § 3.1.1, we may extend the Weil representation $\omega _\psi$ of $\mathrm {U}(X_D) \times \mathrm {U}(Y_D)$ on $\mathcal {S}(Z_+(\mathbb {A}_F))$, where $Z_D = Z_D^+ \oplus Z_D^-$ is an arbitrary polarization, to $R_D(\mathbb {A})$, by using the $\mathrm {U}(X_D) \times \mathrm {U}(Y_D)$-isomorphism $p:\mathcal {S}((X_D^+ \otimes Y_D)(\mathbb {A}))\to \mathcal {S}(Z_+(\mathbb {A}_F))$. Thus for $\phi \in \mathcal {S}(Z_+(\mathbb {A}_F))$, the theta kernel $\theta ^\phi _\psi =\theta ^\phi$ on $R_D(\mathbb {A})$ is defined by

\[ \theta_\psi^{\phi}(g, h) =\theta ^{\phi}(g, h) = \sum _{z_{+} \in Z_D^{+}(F)} \omega_{\psi} (g, h) \phi(z_{+}) \quad \text{for $(g, h) \in R_D(\mathbb{A})$}. \]

Let us define

\[ \mathrm{GU}(X_D, \mathbb{A})^+ = \{h \in \mathrm{GU}(X_D, \mathbb{A}) : \lambda_D(h) \in \nu_D (\mathrm{GU}(Y_D, \mathbb{A}))\} \]

and

\[ \mathrm{GU}(X_D, F)^+ = \mathrm{GU}(X_D, \mathbb{A})^+ \cap \mathrm{GU}(X_D, F). \]

We note that $\nu _D (\mathrm {GU}(Y_D, F_v))$ contains $N_D(D(F_v)^\times )$ for any place $v$. Thus, if $v$ is non-archimedean or complex, we have $\mathrm {GU}(X_D, F_v)^+ = \mathrm {GU}(X_D, F_v)$, and if $v$ is real, $|\mathrm {GU}(X_D, F_v) / \mathrm {GU}(X_D, F_v)^+| \leq 2$.

For a cusp form $f$ on $\mathrm {GU}(X_D, \mathbb {A})^+$, as in § 3.1.1, we define the theta lift of $f$ to $\mathrm {GU}(Y_D, \mathbb {A})$ by

\[ \Theta (f, \phi)(h) := \int_{\mathrm{U}(X_D, F) \backslash \mathrm{U}(X_D, \mathbb{A})} \theta^{\phi}(g_1g, h)f(g_1 g)\,dg_1, \]

where $g \in \mathrm {GU}(X_D, \mathbb {A})^+$ is chosen so that $\lambda _D(g) = \nu _D(h)$. It defines an automorphic form on $\mathrm {GU}(Y_D, \mathbb {A})$. When we regard $\Theta (f, \phi )(h)$ as an automorphic form on $\mathrm {GSU}(Y_D, \mathbb {A})$ by the restriction, we denote it as $\theta (f, \phi )(h)$. For an irreducible cuspidal automorphic representation $(\pi _+, V_{\pi _+})$ of $\mathrm {GU}(X_D, \mathbb {A})^+$, we denote by $\Theta _\psi (\pi _+)$ (respectively, $\theta _\psi (\pi _+)$) the theta lift of $\pi _+$ to $\mathrm {GU}(Y_D, \mathbb {A})$ (respectively, $\mathrm {GSU}(Y_D, \mathbb {A})$), namely

\begin{align*} \Theta_\psi(\pi) &:= \{\Theta (f, \phi) : f \in V_{\pi_+}, \phi \in \mathcal{S}(Z_D^+(\mathbb{A}))\},\\ \theta_\psi(\pi) &:= \{\theta (f, \phi) : f \in V_{\pi_+}, \phi \in \mathcal{S}(Z_D^+(\mathbb{A}))\}, \end{align*}

respectively. Moreover, for an irreducible cuspidal automorphic representation $(\pi, V_\pi )$ of $\mathrm {GU}(X_D, \mathbb {A})$, we define the theta lift $\Theta _\psi (\pi )$ (respectively, $\theta _\psi (\pi )$) of $\pi$ to $\mathrm {GU}(Y_D, \mathbb {A})$ (respectively, $\mathrm {GSU}(Y_D, \mathbb {A})$) by $\Theta _\psi (\pi ) := \Theta _\psi (\pi |_{\mathrm {GU}(X_D, \mathbb {A})^+}\!)$ (respectively, $\theta _\psi (\pi ) := \theta _\psi (\pi |_{\mathrm {GU}(X_D, \mathbb {A})^+}\!)$).

As for the opposite direction, as in § 3.1.1, for a cusp form $f^\prime$ on $\mathrm {GSU}(Y_D, \mathbb {A})$, we define the theta lift of $f^\prime$ to $\mathrm {GU}(X_D, \mathbb {A})^+$ by

\[ \theta (f^\prime, \phi)(g) := \int _{\mathrm{SU}(Y_D, F) \backslash \mathrm{SU}(Y_D, \mathbb{A})} \theta ^{\phi}(g , h_1h)f(h_1 h) \, dh_1 \]

where $h \in \mathrm {GSU}(Y_D, \mathbb {A})$ is chosen so that $\lambda _D(g) = \nu _D(h)$. For an irreducible cuspidal automorphic representation $(\sigma, V_\sigma )$ of $\mathrm {GSU}(Y_D, \mathbb {A})$, we denote by $\theta _\psi (\sigma )$ the theta lift of $\sigma$ to $\mathrm {GU}(X_D, \mathbb {A})^+$. Moreover, we extend $\theta (f^\prime, \phi )$ to an automorphic form on $\mathrm {GU}(X_D, \mathbb {A})$ by the natural embedding

\[ \mathrm{GU}(X_D, F)^+ \backslash \mathrm{GU}(X_D, \mathbb{A})^+ \rightarrow \mathrm{GU}(X_D, F) \backslash \mathrm{GU}(X_D, \mathbb{A}) \]

and extension by zero. Then we define the theta lift $\Theta _\psi (\sigma )$ of $\sigma$ to $\mathrm {GU}(X_D, \mathbb {A})$ as the $\mathrm {GU}(X_D, \mathbb {A})$ representation generated by such $\theta (f^\prime,\phi )$ for $f^\prime \in V_\sigma$ and $\phi \in \mathcal {S} (Z_+(\mathbb {A}))$.

Remark 3.2 Suppose that $(\pi _+, V_{\pi _+})$ (respectively, $(\sigma, V_\sigma )$) is an irreducible cuspidal automorphic representation of $\mathrm {GU}(X_D, \mathbb {A})^+$ (respectively, $\mathrm {GSU}(Y_D, \mathbb {A})$). Suppose moreover that the theta lift $\Theta _\psi (\pi _+)$ (respectively, $\theta _\psi (\sigma )$) is non-zero and cuspidal. Then by Gan [Gan08, Proposition 2.12], $\Theta _\psi (\pi _+)$ (respectively, $\theta _\psi (\sigma )$) is an irreducible cuspidal automorphic representation because of the Howe duality for quaternionic dual pairs proved by Gan and Sun [GS17] and Gan and Takeda [GT16]. We shall study the case $\dim _D~X_D=2$ and $\dim _D~Y_D=3$. In this case, by the conservation relation proved by Sun and Zhu [SZ15], the irreducibility of $\Theta _\psi (\pi _+)$ implies that of $\theta _\psi (\pi _+)$.

3.2.2 Pull-back of the global Bessel periods for the dual pair $(\mathrm {G}_D,\mathrm {GSU}_{3,D})$

The set-up is as follows.

Let $X_D$ be the space of two-dimensional row vectors over $D$ equipped with the hermitian form

\[ (x,x^\prime)_{X_D}=\bar{x}\begin{pmatrix} 0 & 1\\ 1 & 0 \end{pmatrix}\,{}^tx^\prime. \]

Let us take the standard basis of $X_D$ and name the basis vectors as

\[ x_+=(1,0),\quad x_-=(0,1). \]

Then $G_D$ defined by (1.4.2) is the matrix representation of the similitude unitary group $\mathrm {GU}(X_D)$ for $X_D$ with respect to the standard basis.

Let $Y_D$ be the space of three-dimensional column vectors over $D$ equipped with the skew-hermitian form

\[ \langle y, y^\prime \rangle_{Y_D} = {}^{t} y\, \begin{pmatrix} 0 & 0 & \eta\\ 0 & \eta & 0 \\ \eta & 0 & 0 \end{pmatrix}\overline{y^\prime}. \]

Let us take the standard basis of $Y_D$ and name the basis vectors as

\[ y_-={}^t(1,0,0),\quad e={}^t(0,1,0),\quad y_-={}^t(0,0,1). \]

Then $\mathrm {GSU}_{3,D}$ defined in § 2.1.3 is the matrix representation of the group $\mathrm {GSU}(Y_D)$ for $Y_D$ with respect to the standard basis.

We take a polarization $Z_D=Z_{D,+}\oplus Z_{D,-}$ of $Z_D=X_D\otimes _D Y_D$ defined as follows. Let

\[ X_{D,\pm}=x_{\pm}\cdot D, \]

where the double sign corresponds. We decompose $Y_D$ as $Y_D=Y_{D,+}\oplus Y_{D,0}\oplus Y_{D,-}$ where

\[ Y_{D,+}=D\cdot y_+,\quad Y_{D,0}=D\cdot y_0,\quad Y_{D,-}=D\cdot y_-. \]

Then let

(3.2.3)\begin{equation} Z_{D,\pm}=(X_D\otimes Y_{D,\pm})\oplus (X_{D,\pm}\otimes Y_{D,0}), \end{equation}

where the double sign corresponds. To simplify the notation, we denote $z_+ \in Z_{D,+}(\mathbb {A})$ as $z_+=(a,b)$ when

\[ z_+=a\otimes y_+ + b\otimes e\quad \text{where $a\in X_D(\mathbb{A})$ and $b\in X_{D,+} (\mathbb{A})$} \]

and $\phi (z_+)$ as $\phi (a,b)$ for $\phi \in \mathcal {S}(Z_{D,+}(\mathbb {A}))$.

Let us compute the pull-back of the $(X,\chi, \psi )$-Bessel periods on $\mathrm {GSU}_{3,D}$ defined by (2.3.3) with respect to the theta lift from $G_D$.

Proposition 3.2 Let $(\pi _D, V_{\pi _D})$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ whose central character is $\omega _\pi$ and $\chi$ a character of $\mathbb {A}_E^\times$ such that $\chi \mid _{\mathbb {A}^\times }=\omega _\pi ^{-1}$. Let $X\in D^\times$.

Then for $f \in V_{\pi _D}$ and $\phi \in \mathcal {S}(Z_{D, +}(\mathbb {A}))$, we have

(3.2.4)\begin{equation} \mathcal{B}_{X, \chi, \psi}^D(\theta(f: \phi)) = \int_{N_{D}(\mathbb{A}) \backslash G_D^1(\mathbb{A})} B_{\xi_X, \chi^{-1},\psi}(\pi(g)f) (\omega(g, 1) \phi)(v_{D, X}) \, dg, \end{equation}

where

(3.2.5)\begin{equation} \xi_X:=X\eta \bar{X}\in D^-(F),\quad v_{D, X} := (x_-, -\eta^{-1}X x_+)\in Z_{D,+}, \end{equation}

and $B_{\xi _X, \chi ^{-1},\psi }$ denotes the $(\xi _X, \chi ^{-1},\psi )$-Bessel period on $G_D$ defined by (1.4.11).

In particular, the $(\xi _X, \chi ^{-1},\psi )$-Bessel period does not vanish on $V_{\pi _D}$ if and only if the $(X,\chi, \psi )$-Bessel period does not vanish on $\theta _\psi (\pi _D)$.

Proof. The proof of this proposition is similar to that for Proposition 3.1.

Let $N_{0,D}$ be a subgroup of $N_{3,D}$ given by

\[ N_{0, D}(F) = \left\{ u_D(x):=\begin{pmatrix} 1 & 0 & \eta x\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{pmatrix} : x \in F\right\}. \]

Then we note that $N_{0, D}$ is a normal subgroup of $N_{3,D}$ and $\psi _{X, D}$ is trivial on $N_{0, D}(\mathbb {A})$. Since

\[ Z_{D,-}(\mathbb{A})(1, u_D(x)) = Z_{D,-}(\mathbb{A})\quad \text{and}\quad z_+(1, u_D(x)) = z_+ + a \otimes (-\eta x) y_-\quad \text{for $x \in \mathbb{A}$}, \]

we have

\[ (\omega(1, u_D(x)) \phi)(z_+) = \psi \big(- \tfrac{1}{2} \,\mathrm{tr}_D (\langle a, a \rangle \eta^2 x)\big) \phi(z_+). \]

Thus, by an argument similar to that in the proof of Proposition 3.1, one may show that

(3.2.6) \begin{align} & \int_{N_{3,D}(F) \backslash N_{3,D}(\mathbb{A})} \theta(f; \phi)(hu) \psi^{-1}_{X, D}(u) \, du \nonumber\\ &\quad =\int_{N_{3,D}(F) \backslash N_{3,D}(\mathbb{A})}\int_{G_D^1(F) \backslash G_D^1(\mathbb{A})} \sum _{ \overline{a} \in \mathcal{X}_{D,-}}\ \sum_{\gamma \in V_D(a) \backslash G_D^1(F)}\ \sum_{b \in X_{D,+}} \nonumber\\ &\qquad \times (\omega (\gamma g_1 \lambda_s^D(\nu(h)) , uh) \phi)(a, b) f(g_1 \lambda_s(\nu(h))) \, dg_1 \, du. \end{align}

Here $\mathcal {X}_{D,-}$ is the set of equivalence classes $X_{D,-}/ \sim$ where $a\sim a^\prime$ if and only if there exists a $\gamma \in G_D^1(F)$ such that $a^\prime =a\gamma$, $\bar {a}$ denotes the equivalence class of $\mathcal {X}_{D,-}$ containing $a\in X_{D,-}$, and, $V(a)=\{\gamma \in G_D^1(F)\mid a\gamma =a\}$. Then we may rewrite (3.2.6) as

(3.2.7) \begin{align} & \int_{N_{3,D}(F) \backslash N_{3,D}(\mathbb{A})} \theta(f; \phi)(hu) \psi^{-1}_{X, D}(u) \, du\nonumber\\ &\quad =\int_{N_{3,D}(F) \backslash N_{3,D}(\mathbb{A})}\int_{G_D^1(F) \backslash G_D^1(\mathbb{A})} \nonumber\\ &\quad\quad \times \sum_{N_D(F) \backslash G_D^1(F)}\ \sum_{b \in X_{D,+}} (\omega (\gamma g_1 \lambda_s^D(\nu(h)), uh) \phi)(x_{-}, b) f(g_1 \lambda_s(\nu(h))) \, dg_1 \, du. \end{align}

Since, for $u = \bigg (\begin{smallmatrix} 1 & -\eta ^{-1} \bar {A} \eta & B\\ 0 & 1 & A\\ 0 & 0 & 1 \end{smallmatrix}\bigg ) \in N_{3,D}(\mathbb {A})$, we have $Z_{D,-}(\mathbb {A})(1, u) = Z_{D,-}(\mathbb {A})$ and

\begin{align*} z_+ (1, u) &= z_++ x_- \otimes (B^\prime y_- - A e +y_+) +b \otimes (\eta^{-1} \bar{A} \eta y_- + e)\\ &= z_+ + x_- \otimes (B^\prime y_- - A e) +b \otimes (\eta^{-1} \bar{A} \eta y_-) , \end{align*}

we obtain

\[ (\omega(1, u) \phi)(z_+) = \psi (\mathrm{tr}_D (\langle b, x_- \rangle \overline{(e, -Ae)})) \phi(z_+)= \psi (\mathrm{tr}_D (\eta \langle b, x_- \rangle A))\phi(z_+). \]

Hence, in (3.2.7), only $b\in X_{D,-}$ satisfying $\eta \langle b, x_- \rangle = X$, i.e. $b=x_+(-\overline {X}\eta ^{-1} )$ contributes. Thus, our integral is equal to

\begin{align*} & \int_{N_{D}(F) \backslash G_D^1(\mathbb{A})} (\omega (g_1 \lambda_s^D(\nu(h)) , uh) \phi)(v_{D, X}) f(g_1 \lambda_s^D(\nu(h))) \, dg_1 \, du \\ &\quad =\int_{N_{D}(\mathbb{A}) \backslash G_D^1(\mathbb{A})} \int_{N_{D}(F) \backslash N_{D}(\mathbb{A})} \omega (u g_1 \lambda_s^D(\nu(h)) , uh) \phi(v_{D, X})\\ &\qquad \times f( u g_1 \lambda_s^D(\nu(h))) \, dg_1 \, du, \end{align*}

where $v_{D, X} =( x_-, x_+(-\overline {X}\eta ^{-1} ))$. Further, for $u=\big (\begin{smallmatrix}1 & a\\0 & 1\end{smallmatrix}\big ) \in N_D(\mathbb {A})$, we have

\[ (\omega (ug, h) \phi)(v_{D, X})= \psi_{\xi_{X}}(u)^{-1}(\omega(g,h)\phi)(v_{D,X}), \]

where we put $\xi _{X} = X \eta \overline {X}$. Thus, our integral becomes

\begin{align*} & \int_{N_{D}(\mathbb{A}) \backslash G_D^1(\mathbb{A})} \int_{N_{D}(F) \backslash N_{D}(\mathbb{A})} \psi_{\xi_{X}} (u)^{-1} \omega (g_1 \lambda_s^D(\nu(h)) , h) \phi(v_{D, X}) f( u g_1 \lambda_s^D(\nu(h))) \, du \, dg_1 . \end{align*}

As for the integration over $\mathbb {A}^\times M_{X, D}(F) \backslash M_{X, D}(\mathbb {A})$ in (2.3.3), by a direct computation, we see that

\[ \omega (\lambda_s^D(\nu(h)) g_1, h) \phi(v_{D, X}) =|\nu(h)|^{-3} \omega (h_0 \lambda_s(\nu(h)) g_1, 1) \phi(v_{D, X}), \]

where

\[ h = \begin{pmatrix} n_D(h) \cdot (h^X)^\ast & 0 & 0\\ 0 & h & 0\\ 0 & 0 & h^X\end{pmatrix}\quad \text{and} \quad h_0 = \begin{pmatrix} \overline{h^X} & 0\\ 0 & (h^X)^{-1} \end{pmatrix}. \]

Therefore, as in the previous case, we obtain

\[ \mathcal{B}_{X, \chi, \psi}^D(\theta(f: \phi))= \int_{N_{D}(\mathbb{A}) \backslash G_D^1(\mathbb{A})} B_{\xi_X, \chi^{-1},\psi}(\pi(g_1)f) (\omega (g_1, 1) \phi)(v_{D, X}) \, dg_1. \]

The equivalence of the non-vanishing conditions follows from the pull-back formula (3.2.4) as Proposition 3.1.

3.3 Theta correspondence for similitude unitary groups

In our proof of Theorems 1.1 and 1.2, we shall use theta correspondence for similitude unitary groups in addition to theta correspondences for dual pairs $(\mathrm {GSp}_2, \mathrm {GSO}_{4,2})$ and $(G_D, \mathrm {GSU}_{3,D})$. Let us recall the definition of the theta lifts in this case.

Let $(X, (\,{,}\,)_X)$ be an $m$-dimensional hermitian space over $E$, and let $(Y, (\,{,} \,)_Y)$ be an $n$-dimensional skew-hermitian space over $E$. Then we may define the quadratic space

\[ (W_{X, Y}, (\,{,} \,)_{X, Y}) := (\mathrm{Res}_{E / F} X \otimes Y, \mathrm{Tr}_{E / F}((\,{,} \,)_X \otimes \overline{(\,{,}\,)_Y})). \]

This is a $2mn$-dimensional symplectic space over $F$. Then we denote its isometry group by $\mathrm {Sp}(W_{X, Y})$. For each place $v$ of $F$, we denote the metaplectic extension of $\mathrm {Sp}(W_{X, Y})(F_v)$ by $\mathrm {Mp}(W_{X, Y})(F_v)$. In addition, $\mathrm {Mp}(W_{X, Y})(\mathbb {A})$ denotes the metaplectic extension of $\mathrm {Sp}(W_{X, Y})(\mathbb {A})$.

Let $\chi _X$ and $\chi _Y$ be characters of $\mathbb {A}_E^\times / E^\times$ such that $\chi _{X}|_{\mathbb {A}^\times } = \chi _E^m$ and $\chi _{Y}|_{\mathbb {A}^\times } = \chi _E^n$. For each place $v$ of $F$, let

\[ \iota_{\chi_v} : \mathrm{U}(X)(F_v) \times \mathrm{U}(Y)(F_v) \rightarrow \mathrm{Mp}(W_{X, Y}\!)(F_v) \]

be the local splitting given by Kudla [Kud94] depending on the choice of a pair of characters $\chi _v =(\chi _{X,v}, \chi _{Y,v})$. Using this local splitting, we get a splitting

\[ \iota_{\chi} : \mathrm{U}(X)(\mathbb{A}) \times \mathrm{U}(Y)(\mathbb{A}) \rightarrow \mathrm{Mp}(W_{X, Y}\!)(\mathbb{A}), \]

depending on $\chi = (\chi _X, \chi _Y)$. Then by the pull-back, we obtain the Weil representation $\omega _{\psi, \chi }$ of $\mathrm {U}(X)(\mathbb {A}) \times \mathrm {U}(Y)(\mathbb {A})$. When we fix a polarization $W_{X, Y} = W_{X, Y}^+ \oplus W_{X, Y}^-$, we may realize $\omega _{\psi, \chi }$ so that its space of smooth vectors is given by $\mathcal {S}(W_{X, Y}^+(\mathbb {A}))$, the space of Schwartz–Bruhat functions on $W_{X, Y}^+(\mathbb {A})$. We define

\[ R := \{(g, h) \in \mathrm{GU}(X) \times \mathrm{GU}(Y) : \lambda(g) = \lambda(h)\} \supset \mathrm{U}(X)\times \mathrm{U}(Y). \]

Suppose that $\dim Y$ is even and $Y$ is maximally split, in the sense that $Y$ has a maximal isotropic subspace of dimension $\frac {1}{2} \dim Y$. In this case, as in §§ 3.1.1 and 3.2.1, we may extend $\omega _{\psi,\chi }$ to $R(\mathbb {A})$. On the other hand, in this case, we have an explicit local splitting of $R(F_v) \rightarrow \mathrm {Sp}(W_{X, Y})(F_v)$ by Zhang [Zha13] and we may extend $\omega _{\psi,\chi }$ to $R(\mathbb {A})$ using this splitting. These two extensions of $\omega _{\psi,\chi }$ to $R(\mathbb {A})$ coincide.

Then for $\phi \in \mathcal {S}(W_{X, Y}^+(\mathbb {A}))$, we define the theta function $\theta _{\psi, \chi }^\phi$ on $R(\mathbb {A})$ by

(3.3.1)\begin{equation} \theta_{\psi, \chi}^\phi(g, h) = \sum_{w \in W_{X, Y}^+(F)} \omega_{\psi, \chi}(g, h)\phi(w). \end{equation}

Let us define

\begin{align*} \mathrm{GU}(X)(\mathbb{A})^{+}&:= \{g \in \mathrm{GU}(X)(\mathbb{A}) : \lambda(g) \in \lambda(\mathrm{GU}(Y)(\mathbb{A}))\},\\ \mathrm{GU}(X)(F)^{+} &:= \mathrm{GU}(X)(\mathbb{A})^{+} \cap \mathrm{GU}(X)(F). \end{align*}

We define $\mathrm {GU}(Y)(\mathbb {A})^{+}$ and $\mathrm {GU}(Y)(F)^{+}$ in a similar manner. Let $(\sigma, V_\sigma )$ be an irreducible cuspidal automorphic representation of $\mathrm {GU}(X)(\mathbb {A})^{+}$. Then for $\varphi \in V_\sigma$ and $\phi \in \mathcal {S}(W_{X, Y}^+(\mathbb {A}))$, we define the theta lift of $\varphi$ by

\[ \theta_{\psi, \chi}^\phi(\varphi)(h) = \int_{\mathrm{U}(X)(F) \backslash \mathrm{U}(X)(\mathbb{A})} \varphi(g_1g) \theta_{\psi, \chi}^\phi(g_1g, h) \, dg_1, \]

where $g_1 \in \mathrm {GU}(X)(\mathbb {A})^+$ is chosen so that $\lambda (g) = \lambda (h)$. Further, we define the theta lift of $\sigma$ by

\[ \Theta_{\psi, \chi}^{X, Y}(\sigma) = \langle \theta^\phi_{\psi, \chi}(\varphi); \varphi \in \sigma, \phi \in \mathcal{S}(W_{X, Y}^+(\mathbb{A})) \rangle. \]

When the space we consider is clear, we simply write $\Theta ^{X, Y}_{\psi, \chi }(\sigma ) = \Theta _{\psi, \chi }(\sigma )$. Similarly, for an irreducible cuspidal automorphic representation $\tau$ of $\mathrm {U}(Y)(\mathbb {A})$, we define $\Theta _{\psi, \chi }^{Y, X}(\tau )$ and we simply write it by $\Theta _{\psi, \chi }(\tau )$.

4. Proof of the Gross–Prasad conjecture for $(\mathrm {SO}(5),\mathrm {SO}(2))$

In this section we prove Theorem 1.1, i.e. the Gross–Prasad conjecture for $(\mathrm {SO}(5),\mathrm {SO}(2))$, based on the pull-back formulas obtained in the previous section.

4.1 Proof of statement (i) in Theorem 1.1

Let $(\pi, V_\pi )$ be as in Theorem 1.1(i). By the uniqueness of the Bessel model due to Gan, Gross and Prasad [GGP12, Corollary 15.3] at finite places and to Jiang, Sun and Zhu [JSZ10, Theorem A] at archimedean places, there exists uniquely an irreducible constituent $\pi _+^B$ of $\pi \mid _{G_D(\mathbb {A})^+}$ that has the $(\xi, \Lambda, \psi )$-Bessel period.

When $D$ is split and $\pi _+^B$ is a theta lift from an irreducible cuspidal automorphic representation of $\mathrm {GSO}_{3,1}(\mathbb {A})$, our assertion has been proved by Corbett [Cor17]. Hence in the remainder of this subsection, we assume that:

(4.1.1)\begin{align} & \textit{when $D$ is split, $\pi$ is not a theta lift from $\mathrm{GSO}_{3,1}$} \nonumber\\ &\qquad\qquad\qquad\quad \textit{of an irreducible cuspidal automorphic representation}. \end{align}

Let us proceed under the assumption (4.1.1). By Propositions 3.1 and 3.2, the theta lift $\theta _\psi (\pi _+^B)$ of $\pi _+^B$ to $\mathrm {GSU}_{3,D}(\mathbb {A})$ has the $(X_\xi, \Lambda ^{-1}, \psi )$-Bessel period and, in particular, $\theta _\psi (\pi _+^B) \ne 0$ where we take $X_\xi \in D^-(F)$ so that $\xi _{X_\xi } = \xi$. For example, when we take $\xi =\eta$, we may take $X_\xi =1$.

Lemma 4.1 The theta lift $\theta _\psi (\pi _+^B)$ is an irreducible cuspidal automorphic representation of $\mathrm {GSU}_{3,D}(\mathbb {A})$.

Proof. First we note that the irreducibility follows from the cuspidality by Remarks 3.1 and 3.2.

Let us show the cuspidality. Suppose in contrast that $\theta _\psi (\pi _+^B)$ is not cuspidal.

When $D$ is not split, the Rallis tower property implies that the theta lift $\theta _{D, \psi }(\pi _+^B)$ of $\pi _+^B$ to $\mathrm {GSU}_{1,D}( \mathbb {A})$ is non-zero and cuspidal. Let $w$ be a finite place of $F$ such that $D(F_w)$ is split and $\pi _{+,w}^B$ is a generic representation of $G(F_w)^+$. Since $\pi _{+,w}^B$ is generic, the theta lift of $\pi _{+, w}^B$ to $\mathrm {GSO}_2(F_w)$ vanishes by the same argument as for [GRS97, Proposition 2.4]. We note that $\mathrm {GSU}_{1,D}( F_w) \simeq \mathrm {GSO}_2(F_w)$ and, hence, the theta lift of $\pi _+^B$ to $\mathrm {GSU}_{1,D}( \mathbb {A})$ must vanish. This is a contradiction.

Suppose that $D$ is split. Then the theta lift of $\pi _+^B$ to $\mathrm {GSO}_{3,1}$ is non-zero by the Rallis tower property. Moreover, it is not cuspidal by our assumption on $\pi$. Thus, the theta lift of $\pi _+^B$ to $\mathrm {GSO}_{2,0}$ is non-zero, again by the Rallis tower property. Then we reach a contradiction by the same argument as in the non-split case.

We may regard $\theta _\psi (\pi _+^B)$ as an irreducible cuspidal automorphic representation of $\mathrm {PGU}_{2,2}$ or $\mathrm {PGU}_{3,1}$ according to whether $D$ is split or not, under the isomorphism $\Phi$ in (2.2.6) or $\Phi _D$ in (2.2.5). Recall our assumption that $\theta _{\psi _{w}}(\pi _{+,w}^B)$ is generic at a finite place $w$. Then the non-vanishing of $(X_\xi, \Lambda ^{-1}, \psi )$-Bessel period on $\theta _\psi (\pi _+^B)$ implies the non-vanishing of the central value of the standard $L$-function for $\theta _\psi (\pi _+^B)$ of $\mathrm {PGU}_4$ twisted by $\Lambda ^{-1}$, namely

\[ L^{S} \big(\tfrac{1}{2}, \theta_\psi(\pi_+^B) \times \Lambda^{-1} \big) \ne 0 \]

for any finite set $S$ of places of $F$ containing all archimedean places because of the unitary group case of the Gan–Gross–Prasad conjecture for $\theta _\psi (\pi _+^B)$ proved by Proposition A.2 and Remark A.1 in [FM22]. Moreover, from the explicit computation of local theta correspondence in [GT11b, Mor14a], we see that

\[ L(s, \pi_v \times \mathcal{AI} (\Lambda)_v) = L (s, \theta_\psi(\pi_+^B)_v \times \Lambda_v^{-1}) \]

at a finite place $v$ where all data are unramified. Thus, when we take $S_0$, a finite set of places of $F$ containing all archimedean places, so that all data are unramified at $v\notin S_0$, we have

\[ L^{S} \big(\tfrac{1}{2}, \pi \times \mathcal{AI} (\Lambda) \big) = L^{S} \big(\tfrac{1}{2}, \theta_\psi(\pi_+^B) \times \Lambda \big) \ne 0 \]

for any finite set $S$ of places of $F$ with $S\supset S_0$.

Let us show an existence of $\pi ^\circ$. We denote $\theta _\psi (\pi _+^B)$ by $\sigma$. Then the theta lift $\Sigma : = \Theta _{\psi, (\Lambda ^{-1}, \Lambda ^{-1})}(\sigma )$ of $\sigma$ to $\mathrm {GU}_{2,2}$ which we may regard as an automorphic representation of $\mathrm {GSO}_{4,2}$ by the accidental isomorphism (2.2.6), is an irreducible cuspidal globally generic automorphic representation with trivial central character by the proof of [FM22, Proposition A.2] since $\theta _\psi (\pi _+^B)$ has the $(X_\xi, \Lambda ^{-1}, \psi )$-Bessel period.

Here we recall that, by the conservation relation due to Sun and Zhu [SZ15, Theorems 1.10, 7.6], for any irreducible admissible representations $\tau$ of $\mathrm {GO}_{4,2}(k)$ (respectively, $\mathrm {GO}_{3,3}(k)$) over a local field $k$ of characteristic zero, theta lifts of either $\tau$ or $\tau \otimes \det$ to $\mathrm {GSp}_3(k)^+$ (respectively, $\mathrm {GSp}_3(k)$) is non-zero. Thus, we may extend $\Sigma$ to an automorphic representation of $\mathrm {GO}_{4,2}(\mathbb {A})$ as in Harris, Soudry and Taylor [HST93, Proposition 2] so that its local theta lift to $\mathrm {GSp}_3(F_v)^+$ is non-zero at every place $v$.

On the other hand, since $\Sigma$ is nearly equivalent to $\sigma$, we have

(4.1.2)\begin{equation} L^S(s, \Sigma, \mathrm{std}) = L^S(s, \pi, \mathrm{std} \otimes \chi_E) \zeta_F^S(s) \end{equation}

for a sufficiently large finite set $S$ of places of $F$ containing all archimedean places by the explicit computation of local theta correspondences in [GT11b, Mor14a]. Here

\[ L^S(1, \pi, \mathrm{std} \otimes \chi_E) \ne 0 \]

by Yamana [Yam14, Proof of Theorems 10.2, 10.3], since the theta lift $\theta _\psi (\pi _+^B)$ of $\pi _+^B$ to $\mathrm {GSU}_{3,D}( \mathbb {A})$ is non-zero and cuspidal. Hence, the left-hand side of (4.1.2) has a pole at $s=1$. In particular, it is non-zero and the theta lift of $\Sigma$ to $\mathrm {GSp}_3(\mathbb {A})^+$ is non-zero by Takeda [Tak11, Theorem 1.1 (1)]. Further, again by Takeda [Tak11, Theorem 1.1 (1)], this theta lift actually descends to $\mathrm {GSp}_2(\mathbb {A})^+ =G(\mathbb {A})^+$. Namely, the theta lift $\pi ^\prime _+ := \theta _{\psi ^{-1}}(\Sigma )$ of $\Sigma$ to $G(\mathbb {A})^+$ is non-zero since $L^S(s, \Sigma, \mathrm {std})$ actually has a pole at $s=1$.

Suppose that $\pi ^\prime _+$ is not cuspidal. Then by the Rallis tower property, the theta lift of $\Sigma$ to $\mathrm {GL}_2(\mathbb {A})^+$ is non-zero and cuspidal. Meanwhile, the local theta lift of $\Sigma _v$ to $\mathrm {GL}_2(F_v)^+$ vanishes by a computation similar to that for [GRS97, Proposition 3.3] since $\Sigma _v$ is generic. This is a contradiction and, hence, $\pi ^\prime _+$ is cuspidal.

Since $\Sigma$ is generic, so is $\pi ^\prime _+$ by [Mor14a, Proposition 3.3]. Let us take an extension $\pi ^\circ$ of $\pi _+^\prime$ to $G(\mathbb {A})$. Since $|G(F_v) / G(F_v)^+|=2$, we have $\pi ^\prime _{v} \simeq \pi _v$ or $\pi ^\prime _{v} \simeq \pi _v \otimes \chi _{E_v}$ at almost all places $v$ such that $\pi _{+,v}^\prime \simeq \pi _{+, v}^B$. Hence, $\pi$ is locally $G^+$-nearly equivalent to $\pi ^\circ$.

4.2 Some consequences of the proof of Theorem 1.1(i)

As preliminaries for our further considerations, we would like to discuss some consequence of the proof of Theorem 1.1(i) and related results.

First we note the following result concerning the functorial transfer.

Proposition 4.1 Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ with a trivial central character. Assume that there exists a finite place $w$ at which $\pi _w$ is generic and tempered.

Then there exists a globally generic irreducible cuspidal automorphic representation $\pi ^\circ$ of $G(\mathbb {A})$ and an étale quadratic extension $E^\circ$ of $F$ such that $\pi ^\circ$ is $G^{+, E^\circ }$-nearly equivalent to $\pi$. In particular, we have a weak functorial lift of $\pi$ to $\mathrm {GL}_4(\mathbb {A}_{E^\circ })$ with respect to $\mathrm {BC} \circ \mathrm {spin}$.

Moreover, $\pi$ is tempered if and only if $\pi ^\circ$ is tempered.

Remark 4.1 When $D$ is split, our assumption implies that $\pi$ has a generic Arthur parameter. Though our assertion thus follows from the global descent method by Ginzburg, Rallis and Soudry [GRS11] and Arthur [Art13], we shall present another proof which does not refer to these papers.

Proof. Suppose that $D$ is split. When $\pi$ participates in the theta correspondence with $\mathrm {GSO}_{3,1}$, our assertion follows from [Rob01]. Thus, we now assume that the theta lift of $\pi$ to $\mathrm {GSO}_{3,1}$ is zero. By [Li92], $\pi$ has $(S_\circ, \Lambda _\circ, \psi )$-Bessel period for some $S_\circ$ and $\Lambda _\circ$. When $\mathrm {GSO}(S_\circ )$ is not split, the existence of a globally generic irreducible cuspidal automorphic representation follows from Theorem 1.1(i). Suppose that $\mathrm {GSO}(S_\circ )$ is split. Then by Proposition 3.1, the theta lift of $\pi$ to $\mathrm {GSO}_{3,3}$ is non-zero. Since $\pi _w$ is generic, the local theta lift of $\pi _w$ to $\mathrm {GSO}_{1,1}$ is zero as in the proof of Theorem 1.1(i) and, hence, the theta lift of $\pi$ to $\mathrm {GSO}_{1,1}$ is zero. Hence, by the Rallis tower property, either the theta lift of $\pi$ to $\mathrm {GSO}_{2,2}$ or to $\mathrm {GSO}_{3,3}$ is non-zero and cuspidal. Then $\pi$ itself is globally generic by Proposition A.1 in the former case. In the latter case, the global genericity of $\pi$ readily follows from the proof of Soudry [Sou87, Proposition 1.1] (see also Theorem on p. 264 of [Sou87]).

In any case when $D$ is split, we have a globally generic irreducible cuspidal automorphic representation $\pi ^\circ$ of $G(\mathbb {A})$ which is nearly equivalent to $\pi$. Thus, when we take the strong lift of $\pi ^\circ$ to $\mathrm {GL}_4(\mathbb {A})$ by [CKPS04], it is a weak lift of $\pi$ to $\mathrm {GL}_4(\mathbb {A})$.

Suppose that $D$ is not split. Then by Li [Li92], there exist an $\eta _\circ \in D^-(F)$ where $E_\circ :=F(\eta _\circ )$ is a quadratic extension of $F$, and a character $\Lambda _\circ$ of $\mathbb {A}_{E_\circ }^\times / E_\circ ^\times \mathbb {A}^\times$ such that $\pi$ has the $(\eta _\circ, \Lambda _\circ )$-Bessel period. Then there exists a desired automorphic representation $\pi ^\circ$ of $G(\mathbb {A})$ by Theorem 1.1(i).

Let us discuss the temperedness. Denote $\sigma$, $\Sigma$ and $\pi _+^\prime$ as in the proof of Theorem 1.1(i). Suppose that $\pi$ is tempered. Then the temperedness of $\sigma$ follows from a similar argument as in Atobe and Gan [AG17, Proposition 5.5] (see also [GI14, Proposition C.1]) at finite places, from Paul [Pau98, Theorems 15, 30], [Pau05, Theorems 15, 18, Corollary 24] and Li, Paul, Tan and Zhu [LPTZ03, Theorems 4.20, 5.1] at real places and from Adams and Barbasch [AB95, Theorem 2.7] at complex places. Then the temperedness of $\sigma$ implies that of $\Sigma$ by Atobe and Gan [AG17, Proposition 5.5] at finite places, by Paul [Pau00, Theorem 3.4] at non-split real places, by Mœglin [Mœg89, Proposition III.9] at split real places and by Adams and Barbasch [AB95, Theorem 2.6] at complex places. As we obtained the temperedness of $\sigma$ from that of $\pi$, the temperedness of $\Sigma$ implies that of $\pi ^\prime _+$ and, hence, $\pi ^\circ$ is tempered. The opposite direction, i.e. the temperedness of $\pi ^\circ$ implies that of $\pi$, follows by the same argument.

Lemma 4.2 Let $\pi$ be as in Theorem 1.1(i). Suppose that $\sigma =\theta _\psi (\pi _+^B)$ is an irreducible cuspidal automorphic representation of $\mathrm {GSU}_{3,D} (\mathbb {A})$. Here $\pi _+^B$ denotes the unique irreducible constituent of $\pi |_{G_D(\mathbb {A})^+}$ such that $\pi _+^B$ has the $(E, \Lambda )$-Bessel period. We regard $\sigma$ as an automorphic representation of $\mathrm {GU}_{4,\varepsilon }(\mathbb {A})$ via (2.2.5) or (2.2.6) and let $\Pi _\sigma$ denote the base change lift of $\sigma |_{\mathrm {U}_{4,\varepsilon }(\mathbb {A})}$ to $\mathrm {GL}_4(\mathbb {A}_E)$. Let $\pi ^\circ$ be a globally generic irreducible cuspidal automorphic representation of $G(\mathbb {A})$ whose existence is proved in Theorem 1.1(i). We denote the functorial lift of $\pi ^\circ$ to $\mathrm {GL}_4(\mathbb {A})$ by $\Pi _{\pi ^\circ }$.

Suppose that

(4.2.1)\begin{equation} \Pi_{\pi^\circ} = \Pi_{1} \boxplus \cdots \boxplus \Pi_\ell, \end{equation}

where $\Pi _i$ are irreducible cuspidal automorphic representations of $\mathrm {GL}_{n_i}(\mathbb {A})$ and

(4.2.2)\begin{equation} \Pi_{\sigma} =\Pi_1^\prime \boxplus \cdots \boxplus \Pi_k^\prime, \end{equation}

where $\Pi _j^\prime$ are irreducible cuspidal automorphic representations of $\mathrm {GL}_{m_j}(\mathbb {A}_E)$.

Then we have $\Pi _\sigma =\mathrm {BC}(\Pi _{\pi ^\circ })$, $\Pi _{\pi ^\circ }\not \simeq \Pi _{\pi ^\circ }\otimes \chi _E$ and $\mathrm {BC}(\Pi _i)$ is cuspidal for each $i$. In particular, we have $\ell =k$. Here $\mathrm {BC}$ denotes the base change from $F$ to $E$.

Proof. By the explicit computation of local theta correspondences in [GT11b, Mor14a], we see that $(\Pi _{\sigma })_v\simeq \mathrm {BC}(\Pi _{\pi ^\circ } )_v$ at almost all finite places $v$ of $E$. Thus, $\Pi _{\sigma }= \mathrm {BC}(\Pi _{\pi ^\circ } )$ by the strong multiplicity one theorem. In addition, by [CKPS04], we know that $\ell = 1$ or $2$.

Suppose that $\ell =1$. We note that the cuspidality of $\mathrm {BC}(\Pi _{\pi ^\circ })$ is equivalent to $\Pi _{\pi ^\circ } \otimes \chi _E \not \simeq \Pi _{\pi ^\circ }$. Suppose otherwise, i.e. $\Pi _{\pi ^\circ } \simeq \Pi _{\pi ^\circ } \otimes \chi _E$. Then $\Pi _{\pi ^\circ } = \mathcal {AI}(\tau )$ for some irreducible cuspidal automorphic representation $\tau$ of $\mathrm {GL}_2(\mathbb {A}_E)$. Since $\Pi _{\pi ^\circ }$ is a lift from $\mathrm {PGSp}_2$, the central character of $\tau$ needs to be trivial and, hence, $\tau \simeq \tau ^\vee$. On the other hand, we have

\[ \Pi_{\sigma} = \mathrm{BC}(\mathcal{AI}(\tau))= \tau \boxplus \tau^\sigma. \]

Since this is a base change lift of $\sigma |_{\mathrm {U}_{4,\varepsilon }(\mathbb {A})}$, we have $\tau = (\tau ^\sigma )^\vee$ and $\tau \not \simeq \tau ^\sigma$ by [AC89] (see also [PR99, Proposition 3.1]). In particular, $\tau \not \simeq \tau ^\vee$ and we have a contradiction. Thus, $\mathrm {BC}(\Pi _{\pi ^\circ })$ is cuspidal and $k=1$.

Suppose that $\ell =2$. First we show that $\Pi _{\pi ^\circ } \not \simeq \Pi _{\pi ^\circ } \otimes \chi _E$. Suppose otherwise, i.e. $\Pi _{\pi ^\circ } \simeq \Pi _{\pi ^\circ } \otimes \chi _E$. Then either $\Pi _i \simeq \Pi _i \otimes \chi _E$ for $i=1,2$ or $\Pi _2 \simeq \Pi _1 \otimes \chi _E$. In the former case, we have $\Pi _i=\mathcal {AI}(\chi _i)$ with a character $\chi _i$ of $\mathbb {A}_E^\times / E^\times$ for $i=1,2$. Then we have $\Pi _{\pi ^\circ }=\mathcal {AI}(\chi _1)\boxplus \mathcal {AI}(\chi _2)$ and $\Pi _\sigma =\chi _1 \boxplus \chi _1^\sigma \boxplus \chi _2 \boxplus \chi _2^\sigma$. Since $\Pi _{\pi ^\circ }$ is a lift from $\mathrm {PGSp}_2$, the central character of $\mathcal {AI}(\chi _i)$ is trivial and, hence, $\chi _i\mid _{\mathbb {A}^\times }=\chi _E$. On the other hand, since $\Pi _\sigma$ is a base change lift of $\sigma |_{\mathrm {U}_{4,\varepsilon }(\mathbb {A})}$, we see that $\chi _i\mid _{\mathbb {A}^\times }$ is trivial. This is a contradiction. In the latter case, we have $\mathrm {BC}(\Pi _2)=\mathrm {BC}(\Pi _1 \otimes \chi _E) =\mathrm {BC}(\Pi _1)$ and, hence, $\Pi _\sigma =\mathrm {BC}(\Pi _1)\boxplus \mathrm {BC}(\Pi _1)$. This implies that $\Pi _\sigma$ is not in the image of the base change lift from the unitary group and again we have a contradiction. Thus, we have $\Pi _{\pi ^\circ } \not \simeq \Pi _{\pi ^\circ } \otimes \chi _E$. Then $\Pi _i \not \simeq \Pi _i \otimes \chi _E$ at least one of $i=1,2$. Suppose that this is so only for one of the two, say $i=2$. Then $\Pi _1=\mathcal {AI}(\chi )$ for some character $\chi$ of $\mathbb {A}_E^\times / E^\times$ and $\mathrm {BC}(\Pi _2)$ is cuspidal. We have $\Pi _{\pi ^\circ }=\mathcal {AI}(\chi )\boxplus \Pi _2$ and $\Pi _\sigma =\chi \boxplus \chi ^\sigma \boxplus \mathrm {BC}(\Pi _2)$. Then $\chi \mid _{\mathbb {A}^\times }$ is trivial from the former equality and $\chi \mid _{\mathbb {A}^\times }=\chi _E$ from the latter equality as above. Hence, we have a contradiction. Thus, $\mathrm {BC}(\Pi _i)$ for $i=1,2$ are both cuspidal, $\Pi _\sigma =\mathrm {BC}(\Pi _1)\boxplus \mathrm {BC}(\Pi _2)$ and $k=2$.

The following lemma gives the uniqueness of the constant $\ell (\pi )$ defined before Theorem 1.2.

Lemma 4.3 Let $\pi$ be as in Theorem 1.1(i). For $i=1,2$, let $E_i$ be a quadratic extension of $F$ and $\pi ^\circ _i$ an irreducible cuspidal automorphic representation of $G(\mathbb {A})$ which is $G^{+,E_i}$-locally near equivalent to $\pi$. Let $\Pi _{\pi ^\circ _i}$ be the functorial lift of $\pi _i^\circ$ to $\mathrm {GL}_4(\mathbb {A})$ and consider the decomposition

\[ \Pi_{\pi^\circ_i}=\Pi_{i,1}\boxplus\cdots\boxplus\Pi_{i,\ell_i}\quad\text{for $i=1,2$} \]

as (4.2.1). Then we have $\ell _1=\ell _2$.

Proof. Since the case when $E_1=E_2$ is trivial, suppose that $E_1\ne E_2$. Let $K=E_1E_2$. From the definition of the base change, we have

\[ \mathrm{BC}_{K/ E_1} (\mathrm{BC}_{E_1 / F} (\Pi_{\pi^\circ_1})) = \mathrm{BC}_{K / E_1} (\mathrm{BC}_{E_1 / F}(\Pi_{\pi^{\circ}_2})). \]

Hence,

\[ \mathrm{BC}_{E_1 / F} (\Pi_{\pi^\circ_1}) = \mathrm{BC}_{E_1 / F} (\Pi_{\pi^{\circ}_2})\quad \text{or} \quad \mathrm{BC}_{E_1 / F} (\Pi_{\pi^\circ_1}) = \mathrm{BC}_{E_1 / F} (\Pi_{\pi^{\circ}_2})\otimes \chi_{K/ E_1} \]

where $\chi _{K/ E_1}$ denotes the character of $\mathbb {A}_E^\times$ corresponding to $K / E_1$. In the former case, we have

\[ \Pi_{\pi^\circ_1} = \Pi_{\pi^{\circ}_2} \quad \text{or} \quad \Pi_{\pi^\circ_1} = \Pi_{\pi^{\circ}_2} \otimes \chi_{E_1} \]

and our claim follows. In the latter case, since $\chi _{K/ E_1} = \chi _{E_2} \circ N_{E_1 / F}$, we have

\[ \Pi_{\pi^\circ_1} = \Pi_{\pi^{\circ}_2} \otimes \chi_{E_2}\quad \text{or}\quad \Pi_{\pi^\circ_1} = \Pi_{\pi^{\circ}_2} \otimes \chi_{E_2} \chi_{E_1} \]

and our claim follows.

Definition 4.1 Let $\pi$ be as in Theorem 1.1(i). Then we say that $\pi$ is of type I if $\pi$ and $\pi \otimes \chi _E$ are nearly equivalent. Moreover, we say that $\pi$ is of type I-A if $\pi$ participates in the theta correspondence with $\mathrm {GSO}(S_1) =\mathrm {GSO}_{3,1}$ and that $\pi$ is of type I-B if $\pi$ participates in the theta correspondence with $\mathrm {GSO}(X_\circ )$ for some four-dimensional anisotropic orthogonal space $X_\circ$ over $F$ with discriminant algebra $E$.

Remark 4.2 From the proof of Theorem 1.1(i), if $\pi$ is not of type I-A, then the theta lift of $\pi$ to $\mathrm {GSU}_{3,D}$ is cuspidal. Further, we note that $D$ is necessarily split when $\pi$ is of type I-A or I-B, by definition.

In order to study an explicit formula using theta lifts from $G_D(\mathbb {A})$, the following lemma will be important later.

Lemma 4.4 Let $\pi$ be as in Theorem 1.1(i). Then $\pi$ is either type I-A or I-B if and only if $\pi$ is nearly equivalent to $\pi \otimes \chi _E$. In particular, when $\pi$ is neither of type I-A nor I-B, $\pi |_{\mathcal {G}_D}$ is irreducible where

(4.2.3)\begin{equation} \mathcal{G}_D=Z_{G_D}(\mathbb{A})G_D(\mathbb{A})^+ G_D(F). \end{equation}

Proof. Suppose that $\pi$ is nearly equivalent to $\pi \otimes \chi _E$. Then at almost all places $v$ of $F$, $\mathrm {Ind}_{G_D(F_v)^+}^{G_D(F_v)}(\pi _{+,v})$ is irreducible where $\pi _{+,v}$ is an irreducible constituent of $\pi _v\mid _{G_D(F_v)^+}$. This implies that $\pi$ and $\pi ^\circ$ are nearly equivalent and, hence, $\pi ^\circ$ is nearly equivalent to $\pi ^\circ \otimes \chi _E$. Thus, $\Pi _{\pi ^\circ }$ is nearly equivalent to $\Pi _{\pi ^\circ } \otimes \chi _E$ and, hence, $\Pi _{\pi ^\circ } = \Pi _{\pi ^\circ } \otimes \chi _E$ by the strong multiplicity one theorem. When $\pi$ is neither of type I-A nor I-B, this does not happen by Lemmas 4.1 and 4.2.

Suppose that $\pi$ is either of type I-A or I-B. Then $D$ is split and the functorial lift $\Pi _\pi$ of $\pi$ to $\mathrm {GL}_4(\mathbb {A})$ is of the form $\mathcal {AI}(\tau )$ for an irreducible automorphic representation $\tau$ of $\mathrm {GL}_2(\mathbb {A}_E)$ by Roberts [Rob01]. Then we have $\Pi _\pi =\Pi _\pi \otimes \chi _E$. Hence, $\pi$ is nearly equivalent to $\pi \otimes \chi _E$.

When $\pi$ is not nearly equivalent to $\pi \otimes \chi _E$, $\pi \mid _{\mathcal {G}_D}$ is irreducible since $\mathcal {G}_D$ is of index $2$ in $G_D(\mathbb {A})$.

Remark 4.3 This lemma gives a classification of $\pi$ such that the twist $\pi \otimes \chi _E$ of $\pi$ by $\chi _E$ has the same Arthur parameter as $\pi$. A classification of $\pi$ such that $\pi$ and $\pi \otimes \chi _E$ are isomorphic when $G_D \simeq G$ is given in Chan [Cha10].

4.3 Proof of statement (ii) in Theorem 1.1

Suppose that $\pi$ has a generic Arthur parameter.

When there exists a pair $(D^\prime,\pi ^\prime )$ as described in Theorem 1.1(ii), $\pi$ and $\pi ^\prime$ share the same generic Arthur parameter since they are nearly equivalent to each other. Hence, by Theorem 1.1(i), we have

\[ L^S \big(\tfrac{1}{2}, \pi \times \mathcal{AI} (\Lambda)\big) = L^S \big(\tfrac{1}{2}, \pi^\prime \times \mathcal{AI} (\Lambda)\big) \ne 0 \]

when $S$ is a sufficiently large finite set of places of $F$. Then by Remark 1.3, we have

\[ L\big(\tfrac{1}{2},\pi\times\mathcal{AI}(\Lambda)\big)\ne 0, \]

i.e. (1.5.5) holds.

Conversely suppose that $L (\frac {1}{2}, \pi \times \mathcal {AI} (\Lambda )) \ne 0$. There exists an irreducible cuspidal globally generic automorphic representation $\pi ^\circ$ of $G(\mathbb {A})$ which is nearly equivalent to $\pi$ since $\pi$ has a generic Arthur parameter. Let $U$ be a maximal unipotent subgroup of $\mathrm {GSO}_{4,2}$ and $\psi _U$ be a non-degenerate character of $U(\mathbb {A})$ defined below by (6.1.2) and (6.1.3), which are the same as [Mor14a, (2.4)] and [Mor14a, (3.1)], respectively. Let $U_G$ be the maximal unipotent subgroup of $\mathrm {GSp}_{2}$ defined by (6.2.1) and $\psi _{U_G}$ the non-degenerate character of $U_G(\mathbb {A})$ defined by (6.2.2) in § 6.2. Note that in [Mor14a], $U_G$ is denoted by $N$ and $\psi _{U_G}$ is denoted by $\psi _{N}$ in [Mor14a, p. 34] and [Mor14a, (3.2)], respectively. Then we note that the restriction of $\pi ^\circ$ to $G(\mathbb {A})^+$ contains a unique $\psi _{U_G}$-generic irreducible constituent and we denote it by $\pi _+^\circ$. Let us consider the theta lift $\Sigma :=\theta _\psi (\pi _+^\circ )$ of $\pi _+^\circ$ to $\mathrm {GSO}_{4,2}(\mathbb {A})$. Then by [Mor14a, Proposition 3.3], we know that $\Sigma$ is $\psi _U$-globally generic and, hence, non-zero. We divide into two cases according to the cuspidality of $\Sigma$.

Suppose that $\Sigma$ is not cuspidal. Then by Rallis tower property, $\pi ^\circ _+$ participates in the theta correspondence with $\mathrm {GSO}_{3,1}$. As in the proof of Lemma 4.1, the theta lift of $\pi ^\circ _+$ to $\mathrm {GSO}_2$ is zero since $\pi ^\circ _+$ is generic. Hence, the theta lift $\tau :=\theta _\psi ^{X, S_1}(\pi ^\circ _+)$ of $\pi ^\circ _+$ to $\mathrm {GSO}_{3,1}$ is cuspidal and non-zero. By Remark 3.1, $\tau$ is also irreducible.

Recall that

\[ \mathrm{GSO}_{3,1}(F) \simeq \mathrm{GL}_2(E) \times F^\times / \{(z, \mathrm{N}_{E / F}(z)) : z \in E^\times\},\quad \mathrm{PGSO}_{3,1}(F) \simeq \mathrm{PGL}_2(E). \]

Then we may regard $\tau$ as an irreducible cuspidal automorphic representation of $\mathrm {GL}_2(\mathbb {A}_E)$ with a trivial central character since the central character of $\pi ^\circ _+$ is trivial.

Let $\Pi$ denote the strong functorial lift of $\pi ^\circ$ to $\mathrm {GL}_4(\mathbb {A})$ by [CKPS04]. Then at almost all finite places $v$ of $F$, we have $\Pi _v \simeq \mathcal {AI}(\tau )_v$, and thus by the strong multiplicity one theorem, $\Pi = \mathcal {AI}(\tau )$ holds. Since $\pi$ is nearly equivalent to $\pi ^\circ$, Remark 1.3 and our assumption imply that for a sufficiently large finite set $S$ of places of $F$, we have

\begin{align*} L^S \big(\tfrac{1}{2}, \tau \times \Lambda \big) L^S \big(\tfrac{1}{2}, \tau \times \Lambda^{-1} \big) &= L^S \big(\tfrac{1}{2}, \pi^\circ \times \mathcal{AI} (\Lambda)\big)\\ & = L^S \big(\tfrac{1}{2}, \pi \times \mathcal{AI} (\Lambda)\big) \ne 0. \end{align*}

Then by Waldspurger [Wal85], $\tau$ has the split torus model with respect to the character $(\Lambda, \Lambda ^{-1})$. Hence, the equation in Corbett [Cor17, p. 78] implies that $\pi ^\circ$ has the $(E, \Lambda )$-Bessel period. Hence, we may take $D^\prime = \mathrm {Mat}_{2 \times 2}$ and $\pi ^\prime = \pi ^\circ$. Thus, the case when $\Sigma$ is not cuspidal is settled.

Suppose that $\Sigma$ is cuspidal. We may regard $\Sigma$ as an irreducible cuspidal globally generic automorphic representation of $\mathrm {GU}(2,2)$ with trivial central character because of the accidental isomorphism (2.2.6). As in the proof of Theorem 1.1(i), our assumption implies that $L (\frac {1}{2}, \Sigma \times \Lambda ) \ne 0$. Then by [FM22, Proposition A.2], there exists an irreducible cuspidal automorphic representation $\Sigma ^\prime$ of $\mathrm {GU}(V)$ such that $\Sigma ^\prime$ is locally $\mathrm {U}(V)$-nearly equivalent to $\Sigma$ and $\Sigma ^\prime$ has the $(e, \Lambda, \psi )$-Bessel period where $V$ is a $4$-dimensional hermitian space over $E$ whose Witt index is at least $1$. Then we note that $\mathrm {PGU}(V) \simeq \mathrm {PGSO}_{4,2}$ or $\mathrm {P}\mathrm {GU}_{3,D^\prime }$ for some quaternion division algebra $D^\prime$ over $F$.

In the first case, we consider the theta lift $\pi ^\prime _+:=\theta _{\psi ^{-1}}(\Sigma ^\prime )$ of $\Sigma ^\prime$ to $G(\mathbb {A})^+$. Then by the same argument as in the proof of Theorem 1.1(i), we see that $\pi _+^\prime \ne 0$ by Takeda [Tak11, Theorem 1.1 (1)] and that it is an irreducible cuspidal automorphic representation of $G(\mathbb {A})^+$. Since $\Sigma ^\prime$ has the $(e, \Lambda, \psi )$-Bessel period, $\pi ^\prime _+$ has the $(E, \Lambda )$-Bessel period by Proposition 3.1. From the definition, $\pi _+^\prime$ is nearly equivalent to $\pi _+^\circ$. Let us take an irreducible cuspidal automorphic representation $(\pi ^\prime, V_{\pi ^\prime })$ of $G(\mathbb {A})$ such that $\pi ^\prime \mid _{G(\mathbb {A})^+} \supset \pi _+^\prime$. Then $\pi ^\prime$ is locally $G^+$-nearly equivalent, and thus either $\pi ^\prime$ or $\pi ^\prime \otimes \chi _E$ is nearly equivalent to $\pi$ by Remark 1.2. Since both $\pi ^\prime$ and $\pi ^\prime \otimes \chi _E$ have the $(E, \Lambda )$-Bessel period, our claim follows.

In the second case, we consider the theta lift of $\Sigma ^\prime$ to $G_{D^\prime }(\mathbb {A})$. Then by an argument similar to that in the first case, we may show that the theta lift of $\Sigma ^\prime$ to $G_{D^\prime }(\mathbb {A})$ contains an irreducible constituent which is cuspidal, locally $G^+$-nearly equivalent to $\pi$ and has the $(E, \Lambda )$-Bessel period. Here we use [Yam14, Lemma 10.2] and its proof in the case of (${\rm I}_1$) with $n=3, m=2$, noting Remark 4.5. This completes our proof of the existence of a pair $(D^\prime,\pi ^\prime )$.

Let us show the uniqueness of a pair $(D^\prime,\pi ^\prime )$ under the assumption that $\pi$ is tempered. Suppose that for $i=1,2$ there exists a pair $(D_i,\pi _i)$ where $D_i$ is a quaternion algebra over $F$ and $\pi _i$ is an irreducible cuspidal automorphic representation of $G_{D_i}(\mathbb {A})$ which is nearly equivalent to $\pi$ such that $\pi _i$ has the $(E, \Lambda )$-Bessel period.

Suppose that $\pi _i$ is nearly equivalent to $\pi _i \otimes \chi _E$ for $i=1,2$. Then by Lemma 4.4, $\pi _1$, $\pi _2$ are of type I-A or I-B and, in particular, $D_1 \simeq D_2 \simeq \mathrm {Mat}_{2 \times 2}$. Hence, for $i=1,2$, there exist a four-dimensional orthogonal space $X_i$ over $F$ with discriminant algebra $E$ and an irreducible cuspidal automorphic representation $\sigma _i$ of $\mathrm {GSO}(X_i, \mathbb {A})$ such that $\pi _i= \theta _{\psi }(\sigma _i)$. Since $\mathrm {PGSO}(X_i, F) \simeq (D_i^\prime )^\times (E)/ E^\times$ for some quaternion algebra $D_i^\prime$ over $F$, we may regard $\sigma _i$ as an automorphic representation of $(D_i^\prime )^\times (\mathbb {A}_E)$ with the trivial central character. Since $\pi _i$ has the $(E, \Lambda )$-Bessel period, $\sigma _i$ has the split torus period with respect to a character $(\Lambda,\Lambda ^{-1})$ by [Cor17, p. 78]. Hence, $D_i^\prime (E) \simeq \mathrm {Mat}_{2 \times 2}(E)$ by [Wal85]. Since $\sigma _1$ is nearly equivalent to $\sigma _2$, we have $\sigma _1=\sigma _2$ by the strong multiplicity one. Thus, $\pi _1\simeq \pi _2$.

Suppose that $\pi _i$ is neither type I-A nor I-B for $i=1,2$. For each $i$, let us take a unique irreducible constituent $\pi _{i, +}^B$ of $\pi _{i}|_{G_{D_i}(\mathbb {A})^+}$ that has the $(\xi _i, \Lambda, \psi )$-Bessel period. Note that $\pi _{1, +}^B$ and $\pi _{2, +}^B$ are nearly equivalent to each other.

Now let $\sigma _i$ denote the theta lift $\theta _{\psi }(\pi _{i, +}^B)$ of $\pi _{i, +}^B$ to $\mathrm {GSU}_{3,D_i}$. Then we regard $\sigma _i$ as an automorphic representation of $\mathrm {GU}_{4,\varepsilon }$ via (2.2.5), (2.2.6) and let $\Sigma _i := \Theta _{\psi, (\Lambda ^{-1}, \Lambda ^{-1})}$ denote the theta lift of $\sigma _i$ to $\mathrm {GU}_{2,2}$. In turn, we regard $\Sigma _i$ as an automorphic representation of $\mathrm {GSO}_{4,2}$ via (2.2.6) and we denote by $\pi _{i,+}^\prime$ its theta lift to $G(\mathbb {A})^+$. Then from the proof of Theorem 1.1(i), $\sigma _i$, $\Sigma _i$ and $\pi _{i,+}^\prime$ are irreducible and cuspidal. Moreover, $\pi _{1,+}^\prime$ and $\pi _{2,+}^\prime$ are both globally generic and nearly equivalent to each other. Furthermore, since $\pi _i$ is tempered, $\sigma _i=\theta _{\psi }(\pi _{i, +}^B)$ is tempered at finite places by an argument similar to that in Atobe and Gan [AG17, Proposition 5.5] (see also [GI14, Proposition C.1]) and similarly at real and complex places by Paul [Pau98, Theorems 15, 30] and Li, Paul, Tan and Zhu [LPTZ03, Theorems 4.20, 5.1] and, by Adams and Barbasch [AB95, Theorem 2.7], respectively. Similarly $\Sigma _i$ and $\pi _{i,+}^\prime$ are also tempered.

By Propositions 3.1 and 3.2, we know that $\sigma _i$ has the $(X_{\xi _i}, \Lambda, \psi )$-Bessel period. Let $\mathrm {GU}_i$ denote the similitude unitary group which modulo center is isomorphic to $\mathrm {PGSU}_{3, D_i}$ by (2.2.5). Then $\sigma _i\mid _{\mathrm {U}_i}$ has a unique irreducible constituent $\nu _i$ which has the $(X_{\xi _i}, \Lambda, \psi )$-Bessel period. Then by Beuzart-Plessis [Beu16, Beu20] (also by Xue [Xue23] at the real place), we see that $\mathrm {U}_1 \simeq \mathrm {U}_2$ since $\nu _1$ and $\nu _2$ are equivalent to each other. This implies that $D_1 \simeq D_2$ and, hence, $G_{D_1} \simeq G_{D_2}$. Let $D^\prime \simeq D_i$ for $i=1,2$.

We take an irreducible cuspidal automorphic representation $\pi _{i}^\prime$ of $G(\mathbb {A})$ such that $\pi _i^\prime |_{G(\mathbb {A})^+}$ contains $\pi _{i,+}^\prime$. Then by Remark 1.2, we may suppose that $\pi _1^\prime$ is nearly equivalent to $\pi _2^\prime$ or $\pi _2^\prime \otimes \chi _E$. Thus replacing $\pi _2^\prime$ by $\pi _2^\prime \otimes \chi _E$ if necessary, we may assume that $\pi _1^\prime$ and $\pi _2^\prime$ are nearly equivalent to each other. Then since $\pi _1^\prime$ and $\pi _2^\prime$ are generic and they have the same $L$-parameter because of the temperedness of $\pi _i^\prime$, we have $\pi _1^\prime \simeq \pi _2^\prime$ by the uniqueness of the generic member in the $L$-packet by Atobe [Ato17] or Varma [Var17] at finite places and by Vogan [Vog78] at archimedean places. Hence, in particular, $\pi _{1,+}^\prime \simeq \pi _{2,+}^\prime$.

From the definition of $\pi _{+,i}^\prime$, we get $\pi _{1,+}^B \simeq \pi _{2,+}^B$. Then, we see that $\pi _1 \simeq \pi _2 \otimes \omega$ for some character $\omega$ of $G_{D^\prime }(\mathbb {A})$ such that $\omega _v$ is trivial or $\chi _{E,v}$ at each place $v$ of $F$. Since $\pi _1$ and $\pi _2$ have the same $L$-parameter, $\pi _{1,v}$ and $\pi _{1,v}\otimes \omega _v$ are in the same $L$-packet for every place $v$ of $F$.

Let us take a place $v$ of $F$, and write the $L$-parameter of $\pi _{1, v}$ as $\phi _v : WD_{F_v} \rightarrow G^1(\mathbb {C})$. If $\phi _v$ is an irreducible four-dimensional representation, the $L$-packet of $\phi _v$ is singleton and, thus, $\pi _{1, v} \simeq \pi _{2,v}$. Thus, let us suppose that $\phi _{v} = \phi _1 \oplus \phi _2$ with two-dimensional irreducible representations $\phi _i$. Further, we may suppose that $\omega _v = \chi _{E,v}$ since there is nothing to prove when $\omega _v$ is trivial. This implies that $\phi _v \otimes \chi _{E ,v}\simeq \phi _v$. Then, by [PR99, Proposition 3.1], we have $\phi _i=\pi (\chi _i)$ for some character $\chi _i$ of $E_v^\times$ for $i=1,2$. Moreover, any member of the $L$-packet of $\pi _1$ is given by the theta lift from an irreducible representation $\mathrm {JL}(\pi (\chi _1)) \boxtimes \pi (\chi _2)$ of $D^\prime (F_v)^\times \times \mathrm {GL}_2(F_v)$ where $\mathrm {JL}$ denotes the Jacquet–Langlands transfer. Since the theta lift preserves the character twist, we see that

\[ \theta(\mathrm{JL}(\pi(\chi_i)) \boxtimes \pi(\chi_j)) \otimes \chi_{E,v} \simeq \theta(\mathrm{JL}(\pi(\chi_i)) \boxtimes \pi(\chi_j)) \]

by $\pi (\chi _i) \otimes \chi _{E, v} \simeq \pi (\chi _i)$. This shows that in this case, any element in the $L$-packet is invariant under the twist by $\chi _{E,v}$. Thus $\pi _{1, v} \otimes \chi _{E, v} \simeq \pi _{2,v}$ and, hence, $\pi _{1,v} \simeq \pi _{2,v}$.

Remark 4.4 As we remarked in the end of § 1.5, the uniqueness of $(D^\prime, \pi ^\prime )$ follows from the local Gan–Gross–Prasad conjecture for $(\mathrm {SO}(5), \mathrm {SO}(2))$, which is proved by Luo [Luo20] at archimedean places and by Prasad and Takloo-Bighash [PT11] (see also Waldspurger [Wal12b] in general case) at finite places. Our proof gives another proof of the uniqueness.

Remark 4.5 There is a typo in the statement of [Yam14, Lemma 10.2]. The first condition stated there should be the holomorphy at $s=-s_m+\frac {1}{2}$.

5. Rallis inner product formula for similitude groups

In this section, as a preliminary for the proof of Theorem 1.2, we recall Rallis inner product formulas for similitude dual pairs.

5.1 For the theta lift from $G$ to $\mathrm {GSO}_{4,2}$

In this section, we shall recall the Rallis inner product formula for the theta lift from $G$ to $\mathrm {GSO}_{4,2}$. It is derived from the isometry case in a manner similar to that in Gan and Ichino [GI11, § 6], where the case of the theta lift from $\mathrm {GL}_2$ to $\mathrm {GSO}_{3,1}$ is treated.

Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G(\mathbb {A})$ with a trivial central character. Let us define a subgroup $\mathcal {G}$ of $G(\mathbb {A})$ by

(5.1.1)\begin{equation} \mathcal{G} := Z_G(\mathbb{A})G(\mathbb{A})^+ G(F) \end{equation}

and in this section we assume that:

(5.1.2)\begin{equation} \textit{the restriction of $\pi$ to $\mathcal{G}$ is irreducible, i.e. $\pi \otimes \chi_E \not \simeq \pi$} \end{equation}

for our later use.

Let us recall the notation in § 3.1.2. Thus $X$ denotes the four-dimensional symplectic space on which $G$ acts on the right and $Y$ denotes the six-dimensional orthogonal space on which $\mathrm {GSO}_{4,2}$ acts on the left. Then $Z=X\otimes Y$ is a symplectic space over $F$. Here we take $X_\pm \otimes Y$ as the polarization and we realize the Weil representation $\omega _\psi$ of $\mathrm {Mp}(Z)(\mathbb {A})$ on $V_\omega :=\mathcal {S}((X_+ \otimes Y)(\mathbb {A}))$.

Put $X^\Box = X \oplus (-X)$. Then $X^\Box$ is naturally a symplectic space. Let $\widetilde {G}:=\mathrm {GSp}(X^\Box )$ and we denote by $\mathbf {G}$ a subgroup of $G\times G$ given by

\[ \mathbf{G}: = \{(g_1, g_2) \in G \times G: \lambda(g_1) = \lambda(g_2)\}, \]

which has a natural embedding $\iota :\mathbf {G}\to \widetilde {G}$. We define the canonical pairing $\mathcal {B}_\omega : V_\omega \otimes V_\omega \rightarrow \mathbb {C}$ by

\[ \mathcal{B}_\omega(\varphi_1, \varphi_2) := \int_{(X_+ \otimes Y)(\mathbb{A})} \varphi_1(x) \overline{\varphi_2(x)} \, {dx}\quad \text{for $\varphi_1, \varphi_2 \in V_\omega$} \]

where ${dx}$ denotes the Tamagawa measure on $(X_+ \otimes Y)(\mathbb {A})$.

Let $\widetilde {Z} = X^\Box \otimes Y$ and we take a polarization $\widetilde {Z}=\widetilde {Z}_+\oplus \widetilde {Z}_-$ with

\[ \widetilde{Z}_\pm:=(X_\pm\oplus (-X_\pm))\otimes Y, \]

where the double sign corresponds. Let us denote by $\widetilde {\omega }_\psi$ the Weil representation of $\mathrm {Mp}(\widetilde {Z}(\mathbb {A}))$ on $\mathcal {S}( \widetilde {Z}^+(\mathbb {A}))$. On the other hand, let

\[ X^\nabla:=\{(x,-x) : x \in X\}\quad\text{and}\quad \widetilde{X}^\nabla := X^\nabla \otimes Y. \]

Then we have a natural isomorphism

\[ V_\omega \otimes V_\omega\simeq\mathcal{S}( \widetilde{X}^\nabla(\mathbb{A})) \]

by which we regard $\mathcal {S}( \widetilde {X}^\nabla (\mathbb {A}))$ as a representation of $\mathrm {Mp}(Z)(\mathbb {A}) \times \mathrm {Mp}(Z)(\mathbb {A})$. Meanwhile, we may realize $\widetilde {\omega }_\psi$ on $\mathcal {S}(\widetilde {X}^\nabla (\mathbb {A}) )$ and indeed we have an isomorphism

\[ \delta : \mathcal{S}(\widetilde{Z}_+(\mathbb{A})) \rightarrow \mathcal{S}(\widetilde{X}^\nabla (\mathbb{A})) \]

as representations of $\mathrm {Mp}(\widetilde {Z})(\mathbb {A})$ such that

\[ \delta(\varphi_1 \otimes \overline{\varphi}_2)(0) = \mathcal{B}_\omega(\varphi_1, \varphi_2)\quad \text{for $\varphi_1, \varphi_2 \in V_\omega$.} \]

Let us define Petersson inner products on $G(\mathbb {A})$ and $G(\mathbb {A})^+$ as follows. For $f_1, f_2 \in V_\pi$, we define the Petersson inner product $(\, {,}\,)_\pi$ on $G(\mathbb {A})$ by

\[ (f_1, f_2)_\pi := \int_{\mathbb{A}^\times G(F) \backslash G(\mathbb{A})} f_1(g) \overline{f_2(g)} \,dg \]

where $dg$ denotes the Tamagawa measure. Then regarding $f_1, f_2$ as automorphic forms on $G(\mathbb {A})^+$, we define

\[ (f_1, f_2)_{\pi}^+ := \int_{\mathbb{A}^\times G(F)^+ \backslash G(\mathbb{A})^+} f_1(h) \overline{f_2(h)} \,dh, \]

where the measure $dh$ is normalized so that

\[ \mathrm{vol}(\mathbb{A}^\times G(F)^+ \backslash G(\mathbb{A})^+)=1. \]

Then from our assumption (5.1.2) on $\pi$, as in [GI11, Lemma 6.3], we see that

\[ (f_1, f_2)_{\pi}^+= \tfrac{1}{2}(f_1, f_2)_{\pi} \]

since $\mathrm {Vol}(\mathbb {A}^\times G(F) \backslash G(\mathbb {A})) =2$. For each place $v$ of $F$, we take a hermitian $G(F_v)$-invariant local pairing $(\, {,}\,)_{\pi _v}$ of $\pi _v$ so that

(5.1.3)\begin{equation} (f_1,f_2)_\pi = \prod_v(f_{1,v},f_{2,v})_{\pi_v}\quad\text{for $f_i=\otimes_v \,f_{i,v}\in V_\pi\, (i=1,2)$}. \end{equation}

We also choose a local Haar measure $d g_v$ on $G(F_v)$ for each place $v$ of $F$ so that $\mathrm {Vol}(K_{G, v}, d_{g_v})=1$ at almost all $v$, where $K_{G, v}$ is a maximal compact subgroup of $G(F_v)$. We define positive constants $C_{G}$ by

\[ dg = C_{G} \cdot \prod_v dg_v. \]

Local doubling zeta integrals are defined as follows. Let $I(s)$ denote the degenerate principal series representation of $\widetilde {G}(\mathbb {A})$ defined by

\[ I(s): = \mathrm{Ind}_{\widetilde{P}(\mathbb{A})}^{\widetilde{G}(\mathbb{A})} (\chi_E \,\delta_{\widetilde{P}}^{\,s/ 9}), \]

where $\widetilde {P}$ denotes the Siegel parabolic subgroup of $\widetilde {G}$. Then for each place $v$, we define a local zeta integral by

\[ Z_v(s, \Phi_v, f_{1, v}, f_{2,v}) := \int_{G^1(F_v)} \Phi_v(\iota(g_v, 1), s) (\pi_v(g_v)f_{1,v}, f_{2,v})_{\pi_v} \, dg_v \]

for $\Phi _v\in I(s)$, $f_{1,v},f_{2,v}\in V_{\pi _v}$, where $G^1=\{g\in G:\lambda (g)=1\}$. The integral converges absolutely at $s=\frac {1}{2}$ when $\Phi _v \in I_v(s)$ is a holomorphic section by [PR, Proposition 6.4] (see also [GI11, Lemma 6.5]). Moreover, when we define a map $\mathcal {S}( \widetilde {X}^\nabla (\mathbb {A})) \ni \varphi \mapsto [\varphi ]\in I (\frac {1}{2})$ by

\[ [\varphi] \bigg(g, \frac{1}{2} \bigg) := |\nu(g)|^{-4} \bigg(\widetilde{\omega}_\psi\bigg(\begin{pmatrix} 1_4 & \\ & \lambda(g)^{-1}1_4 \end{pmatrix}g\bigg)\varphi\bigg)(0), \]

we may naturally extend $[\varphi ]$ to a holomorphic section in $I(s)$.

By an argument similar to that in the proof of [GI11, Proposition 6.10], we may derive the following Rallis inner product formula in the similitude groups case from the one [GQT14, Theorem 8.1] in the isometry groups case.

Proposition 5.1 Keep the above notation.

Then for decomposable vectors $f = \otimes f_v \in V_\pi$ and $\phi = \otimes \phi _v\in V_\omega$, we have

\begin{align*} \frac{\langle \Theta(f; \phi), \Theta(f; \phi) \rangle}{(f, f)_\pi} &= C_G \cdot\frac{1}{2} \cdot \frac{L(1, \pi, \mathrm{std} \otimes \chi_E)}{L(3, \chi_E) L(2, \mathbf{1}) L(4, \mathbf{1})} \prod_{v}Z_v^\sharp \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg). \end{align*}

Here we recall that $\Theta _\psi (f; \phi )$ is the theta lift of $f$ to $\mathrm {GO}_{4,2}$, $\langle \, {,} \,\rangle$ denotes the Petersson inner product with respect to the Tamagawa measure and we define

\begin{align*} Z_v^\sharp \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg) &:= \frac{1}{(f_v, f_v)_{\pi_v}}\frac{L(3, \chi_{E_v / F_v}) L(2, \mathbf{1}_v) L(4, \mathbf{1}_v) }{L(1, \pi_v, \mathrm{std} \otimes\chi_{E_v / F_v})} Z_v \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg), \end{align*}

which is equal to $1$ at almost all places $v$ of $F$ by [PR].

Recall that $\theta (f; \phi )$ denotes the restriction of $\Theta _\psi (f; \phi )$ to $\mathrm {GSO}_{4,2}(\mathbb {A})$, namely the theta lift of $f$ to $\mathrm {GSO}_{4,2}$. Then as in [GI11, Lemma 2.1], we see that

\[ 2\langle \Theta(f; \phi), \Theta(f; \phi) \rangle = \langle \theta(f; \phi), \theta(f; \phi) \rangle, \]

where the right-hand side denotes the Petersson inner product on $\mathrm {GSO}_{4,2}$ with respect to the Tamagawa measure. Hence, Proposition 5.1 yields

(5.1.4) \begin{align} \frac{\langle \theta(f; \phi), \theta(f; \phi) \rangle}{(f, f)_{\pi}} &= C_G \cdot \frac{L(1, \pi, \mathrm{std} \otimes \chi_E)}{L(3, \chi_E) L(2, \mathbf{1}) L(4, \mathbf{1}) } \prod_{v} Z_v^\sharp \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg). \end{align}

5.2 Theta lift from $G_D$ to $\mathrm {GSU}_{3,D}$

In this subsection, we shall consider the Rallis inner product formula for the theta lift from $G_D$ to $\mathrm {GSU}_{3,D}$ as in the previous section. We recall that the formula in the case of isometry groups is proved by Yamana [Yam14, Lemma 10.1] where our case corresponds to (${\rm I}_3$) with $m=3, n=2$.

Let $(\pi, V_\pi )$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ with a trivial central character. Recall that $\mathcal {G}_D$ denotes the subgroup of $G_D(\mathbb {A})$ given by (4.2.3). In this section, assume that

(5.2.1)\begin{equation} \textit{the restriction of $\pi$ to $\mathcal{G}_D$ is irreducible} \end{equation}

for our later use.

Let us recall the notation in § 3.2.2. Thus, $X_D$ denotes the hermitian space of degree two over $D$ on which $G_D$ acts on the right and $Y_D$ denotes the skew-hermitian space of degree three over $D$ on which $\mathrm {GSU}_{3,D}$ acts on the left. Then $Z_D= X_D \otimes _D Y_D$ is a symplectic space over $F$ by (3.2.1). Here we take $X_{D,\pm }\otimes _D Y_D$ as the polarization and we realize the Weil representation $\omega _\psi$ of $\mathrm {Mp}(Z_D)(\mathbb {A})$ on $V_{\omega,D}:=\mathcal {S}((X_{D,+}\otimes _D Y_D)(\mathbb {A}))$.

Put $X_D^\Box = X_D \oplus \overline {X_D}$. Then $X_D^\Box$ is naturally a hermitian space over $D$. Let $\widetilde {G}_D:=\mathrm {GU}(X_D^\Box )$ and we denote by $\mathbf {G}_D$ a subgroup of $G_D\times G_D$ given by

\[ \mathbf{G}_D := \{(g_1, g_2) \in G_D \times G_D : \lambda(g_1) = \lambda(g_2)\}, \]

which has a natural embedding $\iota :\mathbf {G}_D\to \widetilde {G}_D$. We define the canonical pairing $\mathcal {B}_\omega : V_{\omega,D} \otimes V_{\omega,D}\rightarrow \mathbb {C}$ by

\[ \mathcal{B}_\omega(\varphi_1, \varphi_2) := \int_{(X_{D,+} \otimes Y_D)(\mathbb{A})} \varphi_1(x) \,\overline{\varphi_2(x)} \, {dx}\quad\text{for $\varphi_1, \varphi_2 \in V_{\omega,D}$}, \]

where ${dx}$ denotes the Tamagawa measure on $(X_{D,+} \otimes Y_D)(\mathbb {A})$.

Let $\widetilde {Z}_D = X_D^\Box \otimes Y_D$ and we take a polarization $\widetilde {Z}_D=\widetilde {Z}_{D,+}\oplus \widetilde {Z}_{D,-}$ with

\[ \widetilde{Z}_{D,\pm}=(X_{D,\pm}\oplus \overline{-X_{D,\pm}})\otimes Y_D, \]

where the double sign corresponds. Let us denote by $\widetilde {\omega }_\psi$ the Weil representation of $\mathrm {Mp}(\widetilde {Z}_D)(\mathbb {A})$ on $\mathcal {S}( \widetilde {Z}_{D,+}(\mathbb {A}))$. On the other hand, let

\[ X_D^\nabla:=\{(x,\overline{x}) : x \in X_D\}\quad\text{and}\quad \widetilde{X}_D^\nabla := X_D^\nabla \otimes Y_D . \]

Then we have a natural isomorphism

\[ V_{\omega,D} \otimes V_{\omega,D}\simeq\mathcal{S}( \widetilde{X}_D^\nabla(\mathbb{A})) \]

by which we regard $\mathcal {S}( \widetilde {X}_D^\nabla (\mathbb {A}))$ as a representation of $\mathrm {Mp}(Z_D)(\mathbb {A}) \times \mathrm {Mp}(Z_D)(\mathbb {A})$. Meanwhile, we may realize $\widetilde {\omega }_\psi$ on $\mathcal {S}(\widetilde {X}_D^\nabla (\mathbb {A}) )$ and indeed we have an isomorphism

\[ \delta : \mathcal{S}(\widetilde{Z}_{D,+}(\mathbb{A})) \rightarrow \mathcal{S}( \widetilde{X}_D^\nabla (\mathbb{A})) \]

as representations of $\mathrm {Mp}(\widetilde {Z}_D)(\mathbb {A})$ such that

\[ \delta(\varphi_1 \otimes \overline{\varphi}_2)(0) = \mathcal{B}_\omega(\varphi_1, \varphi_2)\quad\text{for $\varphi_1, \varphi_2 \in V_{\omega,D}$.} \]

Let us define Petersson inner products on $G_D(\mathbb {A})$ and $G_D(\mathbb {A})^+$ as follows. For $f_1, f_2 \in V_{\pi _D}$, we define the Petersson inner product $(\, {,}\,)_{\pi _D}$ on $G_D(\mathbb {A})$ by

\[ (f_1, f_2)_{\pi_D}: = \int_{\mathbb{A}^\times G_D(F) \backslash G_D(\mathbb{A})} f_1(g) \,\overline{f_2(g)} \,dg, \]

where $dg$ denotes the Tamagawa measure. Then regarding $f_1, f_2$ as automorphic forms on $G_D(\mathbb {A})^+$, we define

\[ (f_1, f_2)_{\pi_D}^+ := \int_{\mathbb{A}^\times G_D(F)^+ \backslash G_D(\mathbb{A})^+} f_1(h) \,\overline{f_2(h)} \,dh, \]

where the measure $dh$ is normalized so that

\[ \mathrm{vol}(\mathbb{A}^\times G_D(F)^+ \backslash G_D(\mathbb{A})^+)=1. \]

Then from our assumption (5.2.1) on $\pi _D$, as in [GI11, Lemma 6.3], we see that

\[ (f_1, f_2)_{\pi_D}^+= \tfrac{1}{2} (f_1, f_2)_{\pi_D} \]

since $\mathrm {Vol}(\mathbb {A}^\times G_D(F) \backslash G_D(\mathbb {A})) =2$. For each place $v$ of $F$, we take a hermitian $G_D(F_v)$-invariant local pairing $(\, {,}\,)_{\pi _{D,v}}$ of $\pi _{D,v}$ so that

(5.2.2)\begin{equation} (f_1,f_2)_{\pi_D} = \prod_v(f_{1,v},f_{2,v})_{\pi_{D,v}} \quad\text{for $f_i=\otimes_v \,f_{i,v}\in V_{\pi_D}\ (i=1,2)$}. \end{equation}

As in the previous section, we choose local Haar measures $dg_v$ on $G_D(F_v)$ at each place $v$ of $F$ and we have

\[ dg = C_{G_D} \cdot \prod_v dg_v \]

for some positive constant $C_{G_D}$.

Local doubling zeta integrals are defined as follows. Let $I_D(s)$ denote the degenerate principal series representation of $\widetilde {G}_D(\mathbb {A})$ defined by

\[ I_D(s) := \mathrm{Ind}_{\widetilde{P}_D(\mathbb{A})}^{\widetilde{G}_D(\mathbb{A})} (\chi_E \,\delta_{\widetilde{P}_D}^{\,s / 9}), \]

where $\widetilde {P}_D$ denotes the Siegel parabolic subgroup of $\widetilde {G}_D$. Then for each place $v$, we define a local zeta integral for $\Phi _v\in I_{D,v}(s)$, $f_{1,v},f_{2,v}\in V_{\pi _{D,v}}$ by

\[ Z_v(s, \Phi_v, f_{1, v}, f_{2,v}) := \int_{G^1_{D}(F_v)} \Phi_v(\iota(g_v, 1), s) (\pi_{D,v}(g_v)f_{1,v}, f_{2,v})_{\pi_v} \, dg_v, \]

where $G_D^1=\{g\in G_D:\lambda (g)=1\}$. The integral converges absolutely at $s=\frac {1}{2}$ when $\Phi _v \in I_{D,v}(s)$ is a holomorphic section by [PR, Proposition 6.4] (see also [GI11, Lemma 6.5]). Moreover, when we define a map $\mathcal {S}( \widetilde {X}_D^\nabla (\mathbb {A})) \ni \varphi \mapsto [\varphi ]\in I_D (\frac {1}{2})$ by

\[ [\varphi] \bigg(g, \frac{1}{2} \bigg): = |\lambda(g)|^{-4} \bigg(\widetilde{\omega}_\psi\bigg(\begin{pmatrix}1_4 & \\ & \lambda(g)^{-1}1_4\end{pmatrix}g\bigg)\varphi\bigg)(0), \]

we may naturally extend $[\varphi ]$ to a holomorphic section in $I_D(s)$.

By an argument similar to that in the proof of [GI11, Proposition 6.10], we may derive the following Rallis inner product formula in the similitude groups case from that [Yam13, Theorem 2] in the isometry groups case.

Proposition 5.2 Keep the above notation.

Then for decomposable vectors $f = \otimes f_v \in V_{\pi _D}$ and $\phi = \otimes \phi _v\in V_{\omega,D}$, we have

\[ \frac{\langle \theta(f; \phi), \theta(f; \phi) \rangle}{(f_{\pi_D}, f_{\pi_D})} = \frac{L(1, \pi, \mathrm{std} \otimes \chi_E)}{L(3, \chi_E) L(2, \mathbf{1}) L(4, \mathbf{1})} \prod_{v} Z_v^\sharp \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg). \]

Here recall that $\theta _\psi (f; \phi )$ is the theta lift of $f$ to $\mathrm {GSU}_{3,D}$, $\langle \, {,}\,\rangle$ denotes the Petersson inner product with respect to the Tamagawa measure and we define

\begin{align*} Z_v^\sharp \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg) &:= \frac{1}{(f_v, f_v)_{\pi_{D, v}}}\frac{L(3, \chi_{E_v / F_v}) L(2, \mathbf{1}_v) L(4, \mathbf{1}_v) }{L(1, \pi_v, \mathrm{std} \otimes\chi_{E_v / F_v})}\\ &\quad \times Z_v \bigg(\frac{1}{2}, [\delta(\phi_v \otimes \phi_v)], f_v, f_v \bigg), \end{align*}

which is equal to $1$ at almost all places $v$ of $F$ by [PR].

6. Explicit formula for Bessel periods on $\mathrm {GU}(4)$

Let $\mathrm {GU}(4)$ stand for one of $\mathrm {GU}_{2,2}$ or $\mathrm {GU}_{3,1}$. In [FM22], the explicit formula for the Bessel periods on $\mathrm {GU}(4)$ is proved under the assumption that the explicit formula for the Whittaker periods on $\mathrm {GU}_{2,2}$ holds. In this section we shall show that this assumption is indeed satisfied in the cases we need, from the explicit formula for the Whittaker periods on $G=\mathrm {GSp}_2$, which, in turn, will be proved in Appendix A. Thus, the explicit formula for the Bessel periods on $\mathrm {GU}(4)$ holds by [FM22], in the cases which we need for the proof of Theorem 1.2.

6.1 Explicit formulas

Let $(\pi, V_{\pi })$ be an irreducible cuspidal tempered globally generic automorphic representation of $G(\mathbb {A})$ such that $\pi |_{\mathcal {G}}$ is irreducible. We recall that the subgroup $\mathcal {G}$ of $G(\mathbb {A})$ is defined by (5.1.1). Let $\pi ^\circ$ denote the unique generic irreducible constituent of $\pi |_{G(\mathbb {A})^+}$. Let $(\Sigma, V_\Sigma )$ denote the theta lift of $\pi ^\circ$ to $\mathrm {GSO}_{4,2}(\mathbb {A})$. Then as in [Mor14a, Proposition 3.3], we know that $\Sigma$ is an irreducible globally generic cuspidal tempered automorphic representation. Here we prove the explicit formula for the Whittaker periods for $\Sigma$ assuming the explicit formula for the Whittaker periods for $\pi$.

Let us recall some notation. Let $X, Y, Y_0$ and $Z$ be as in § 3.1.2 and we use a polarization $Z = Z_+ \oplus Z_-$ with

\[ Z_{\pm} = (X \otimes Y_{\pm}) \oplus (X_{\pm} \otimes Y_0), \]

where the double sign corresponds. We write $z_+=(a_1, a_2; b_1, b_2)$ when

\[ z_+=a_1 \otimes y_1 + a_2 \otimes y_2 + b_1 \otimes e_1+b_2\otimes e_2\in Z_+\quad \text{with $a_i \in X$, $b_i \in X_+$}. \]

Recall that the unipotent subgroups $N_0$, $N_1$ and $N_2$ of $\mathrm {GSO}_{4,2}$ are defined by (3.1.10), (3.1.11) and (3.1.12), respectively. Let us define an unipotent subgroup $\tilde {U}$ of $\mathrm {GSO}_{4,2}$ by

(6.1.1)\begin{equation} \tilde{U} := \left\{\tilde{u}(b):= \begin{pmatrix} 1 & -{}^{t}\widetilde{X} S_1 & 0\\ 0 & 1_4 & \widetilde{X}\\ 0 & 0 & 1 \end{pmatrix}: \widetilde{X} = \begin{pmatrix} 0\\0\\0\\-b\end{pmatrix}\right\}, \end{equation}

where $S_1$ is given by (2.1.2). Let

(6.1.2)\begin{equation} U:=N_{4,2}\tilde{U}. \end{equation}

Then $U$ is a maximal unipotent subgroup of $\mathrm {GSO}_{4,2}$ and we have

\[ N_0 \triangleleft N_0 N_1 \triangleleft N_0 N_1 N_2= N_{4,2} \triangleleft N_{4,2}\tilde{U} = U. \]

Then we define a non-degenerate character $\psi _U$ of $U(\mathbb {A})$ by

(6.1.3)\begin{equation} \psi_U(u_0(x)u_1(s_1, t_1)u_2(s_2, t_2)\tilde{u}(b)) := \psi(2dt_2+b). \end{equation}

By [Mor14a, Proposition 3.3], $\Sigma$ is $\psi _U$-generic. Namely

\[ W^{\psi_U}(\varphi) := \int_{U(F) \backslash U(\mathbb{A})} \varphi(u) \psi_U^{-1}(u)\,du \quad\text{for $ \varphi \in V_\Sigma$},\label{W U} \]

is not identically zero on $V_\Sigma$. Now we regard $\Sigma$ as an automorphic representation of $\mathrm {GU}_{2,2}$ by the accidental isomorphism (2.2.6) and let $\Pi _{\Sigma } =\Pi _{1}^\prime \boxplus \cdots \boxplus \Pi _\ell ^\prime$ denote the base change lift of $\Sigma \mid _{\mathrm {U}_{2,2}}$ to $\mathrm {GL}_4(\mathbb {A}_E)$ where $\Pi _i^\prime$ is an irreducible cuspidal automorphic representations of $\mathrm {GL}_{m_i}(\mathbb {A}_E)$. Here the existence of $\Pi _\Sigma$ follows from [KMSW14].

Recall that in § 5.1, the Petersson inner products on $G(\mathbb {A})$ and $\mathrm {GSO}_{4,2}(\mathbb {A})$ using the Tamagawa measures, denoted respectively as $(\, {,}\,)$ and $\langle \, {,}\,\rangle$, are introduced. Moreover at each place $v$ of $F$, we choose and fix an $G(F_v)$-invariant hermitian inner product $(\,{,}\,)_v$ on $V_{\pi ^\circ _v}$ so that the decomposition formula (5.1.3) holds. Similarly at each place $v$, we choose and fix a $\mathrm {GSO}_{4,2}(F_v)$-invariant hermitian inner product $\langle \,{,}\,\rangle _v$ on $V_{\Sigma _v}$ so that the decomposition formula

(6.1.4)\begin{equation} \langle \phi_1,\phi_2\rangle=\prod_v \langle \phi_{1,v},\phi_{2,v}\rangle_v\quad \text{for $\phi_i=\otimes_v \phi_{i,v}\in V_{\Sigma}\quad(i=1,2)$} \end{equation}

holds.

Then as in § 2.4, at each place $v$ of $F$, we may define a local period $\mathcal {W}_v(\varphi _v)$ for $\varphi _v \in V_{\Sigma _v}$ by the stable integral

(6.1.5)\begin{equation} \mathcal{W}_v(\varphi_v):=\int_{U(F_v)}^{\mathrm{st}} \frac{\langle \Sigma_v(n_v)\varphi_v, \varphi_v \rangle_v}{{\langle \varphi_v, \varphi_v \rangle_v}}\cdot \psi_U^{-1}(n_v)\, dn_v \end{equation}

when $v$ is finite. When $v$ is archimedean, we use the Fourier transform to define $\mathcal {W}_v(\varphi _v)$. See [Liu16, Propositions 3.5, 3.15] for the details.

We shall prove the following theorem, namely the explicit formula for the Whittaker periods on $V_\Sigma$, in § 6.2.

Theorem 6.1 For a non-zero decomposable vector $\varphi = \otimes \varphi _v \in V_\Sigma$, we have

(6.1.6)\begin{equation} \frac{|W^{\psi_U}(\varphi)|^2}{ \langle \varphi, \varphi \rangle} = \frac{1}{2^\ell}\cdot \frac{\prod_{j=1}^4 L (j, \chi_{E}^j)}{L (1, \Pi_{\Sigma}, \mathrm{As}^+)} \cdot \prod_v \mathcal{W}_v^\circ(\varphi_v), \end{equation}

where

\[ \mathcal{W}_v^\circ(\varphi_v):=\frac{L(1,\Pi_{\Sigma_v}, \mathrm{As}^+)}{ \prod_{j=1}^4 L (j, \chi_{E_v}^j)}\cdot \mathcal{W}_v(\varphi_v). \]

Here we note that $\mathcal {W}_v^\circ (\varphi _v) = 1$ at almost all places $v$ by Lapid and Mao [LM15].

Before proceeding to the proof of Theorem 6.1, by assuming it, we prove the following theorem, namely the explicit formula for the Bessel periods on $\mathrm {GU}(4)$.

Theorem 6.2 Let $(\pi, V_\pi )$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with trivial central character. Suppose that $\pi$ has the $(\xi,\Lambda,\psi )$-Bessel period and that $\pi$ is neither of type I-A nor type I-B. Let $\pi _+^B$ denote the unique irreducible constituent of $\pi |_{G_D(\mathbb {A})^+}$ which has the $(\xi,\Lambda,\psi )$-Bessel period. We denote by $(\sigma, V_\sigma )$ the theta lift of $\pi _+^B$ to $\mathrm {GSU}_{3,D}$, which is an irreducible cuspidal automorphic representation by Lemmas 4.1 and 4.4.

Then for a non-zero decomposable vector $\varphi = \otimes \varphi _v \in V_\sigma$, we have

\[ \frac{|\mathcal{B}_{X, \psi. \Lambda}(\varphi)|^2}{(\varphi, \varphi)} =\frac{1}{2^\ell} \bigg(\prod_{j=1}^4 L(1, \chi_E^j)\bigg) \frac{L (\frac{1}{2}, \sigma \times \Lambda^{-1})}{L(1, \pi, \mathrm{std} \otimes \chi_E) L(1, \chi_E)} \prod_v \alpha_{\Lambda_v, \psi_{X, v}}^{\natural}(\varphi_v), \]

where

\[ \alpha_{\Lambda_v, \psi_{X, v}}^{\natural}(\varphi_v) = \bigg(\prod_{j=1}^4 L(1, \chi_{E, v}^j)\bigg)^{-1} \frac{L(1, \pi_v, \mathrm{std} \otimes \chi_{E,v}) L(1, \chi_{E_v})}{L (\frac{1}{2}, \sigma_v \times \Lambda_v^{-1})}\cdot \frac{\alpha_{\Lambda_v, \psi_{X, v}}(\varphi_v)}{(\varphi_v, \varphi_v)_v} \]

and $X\in D^\times$ is taken so that $\xi =S_X$ in (3.2.5).

Proof. Let us regard $\sigma$ as an automorphic representation of $\mathrm {GU}(4)$ with trivial central character via the accidental isomorphisms $\Phi$ (2.2.6) or $\Phi _D$ (2.2.5), depending whether $D$ is split or not. Let $\theta (\sigma ) = \Theta _{\psi, (\Lambda ^{-1},\Lambda ^{-1})}(\sigma )$ denote the theta lift of $\sigma$ to $\mathrm {GU}_{2,2}$ with respect to $\psi$ and $(\Lambda ^{-1}, \Lambda ^{-1})$. By [FM22, Proposition 3.1], $\theta (\sigma )$ is globally generic and, in particular, non-zero. By the same argument as in the proof of [FM22, Theorem 1], we see that $\theta (\sigma )$ is cuspidal and hence irreducible by Remarks 3.1 and 3.2. Moreover by the unramified computations in [Kud86] and [Mor14a, (3.6)], we see that $L^S(s, \Sigma, \wedge _t^2)$ has a pole at $s=1$ when $S$ is a sufficiently large finite set of places of $F$ containing all archimedean places, where $L^S(s, \Sigma, \wedge _t^2)$ denotes the twisted exterior square $L$-function of $\Sigma$ (see [FM13b, § 2.1.1] for the definition). Since $\theta (\sigma )$ is generic, [FM13b, Theorem 4.1] implies that it has the unitary Shalika period defined in [FM13b, (2.5)]. Then, by [Mor14a, Theorem B], the theta lift of $\theta (\sigma )$ to $G(\mathbb {A})^+$, which we denote by $(\pi _{+}^\prime, V_{\pi _{+}^\prime })$, is an irreducible cuspidal globally generic automorphic representation of $G(\mathbb {A})^+$. We note that $\pi _+^B$ is nearly equivalent to $\pi _+^\prime$.

Let us take an irreducible cuspidal automorphic representation $(\pi ^\prime, V_{\pi ^\prime })$ of $G(\mathbb {A})$ such that $V_{\pi ^\prime }|_{G(\mathbb {A})^+} \supset V_{\pi _+^\prime }$. Then $\pi ^\prime$ is globally generic. Moreover, $\pi ^\prime \otimes \chi _E$ is not nearly equivalent to $\pi ^\prime$ by our assumption on $\pi$. Hence, $\pi ^\prime |_{\mathcal {G}}$ is irreducible. Thus, we may apply Theorem 6.1, taking $\pi ^\circ = \pi ^\prime$ and $\Sigma = \theta (\sigma )$, and we obtain the explicit formula for the Whittaker periods on $\theta (\sigma )$. Then by [FM22, Theorem A.1], the required explicit formula for the Bessel periods follows.

6.2 Proof of Theorem 6.1

We reduce Theorem 6.1 to a certain local identity in § 6.2.2 and then prove the local identity in § 6.2.3.

As we stated in the beginning of this section, what we do essentially is to deduce the explicit formula (6.1.6) for the Whittaker periods on $\mathrm {GSO}_{4,2}$ from (6.2.3) below, the one for the Whittaker periods on $G$.

6.2.1 Explicit formula for the Whittaker periods on $G=\mathrm {GSp}_2$

Let $U_G$ denote the maximal unipotent subgroup of $G$. Namely

(6.2.1)\begin{equation} U_G := \bigg\{m(n)\begin{pmatrix}1_2 & X\\ 0 & 1_2 \end{pmatrix} : X \in \mathrm{Sym_2}, n \in N_2\bigg\}, \end{equation}

where $m(h)=\big (\begin{smallmatrix} h & 0 \\0 & {}^{t}h^{-1}\end{smallmatrix}\big )$ for $h\in \mathrm {GL}_2$ and $N_2$ denotes the group of upper unipotent matrices in $\mathrm {GL}_2$. Then we define a non-degenerate character $\psi _{U_G}$ of $U_G(\mathbb {A})$ by

(6.2.2)\begin{equation} \psi_{U_G}(u) := \psi(u_{1\,2}+d \,u_{2\,4})\quad\text{for $u=(u_{i\,j})\in U_G(\mathbb{A})$}. \end{equation}

Then for an automorphic form $\phi$ on $G(\mathbb {A})$, we define the Whittaker period $W_{\psi _{U_G}}(\phi )$ of $\phi$ by

\[ W_{\psi_{U_G}}(\phi) = \int_{U_G(F) \backslash U_G(\mathbb{A})} \phi(n) \,\psi_{U_G}^{-1}(n) \,dn.\label{W U_G} \]

The following theorem shall be proved in Appendix A.

Theorem 6.3 Suppose that $(\pi,V_{\pi })$ is an irreducible cuspidal tempered globally generic automorphic representation of $G(\mathbb {A})$. Let $\Pi _{\pi } = \Pi _1 \boxplus \cdots \boxplus \Pi _k$ denote the functorial lift of $\pi$ to $\mathrm {GL}_4(\mathbb {A})$.

Then for any non-zero decomposable vector $\varphi = \otimes \varphi _v \in V_{\pi }$, we have

(6.2.3)\begin{equation} \frac{|W_{\psi_{U_G}}(\varphi)|^2}{(\varphi, \varphi)} = \frac{1}{2^k}\cdot \frac{\prod_{j=1}^2\xi_F(2j)}{L(1, \Pi_{\pi}, \mathrm{Sym}^2)} \cdot \prod_v \mathcal{W}_{G,v}^\circ(\varphi_v). \end{equation}

Here $\mathcal {W}^\circ _{G, v}(\varphi _v)$ is defined by

\[ \mathcal{W}^\circ_{G, v}(\varphi_v) =\frac{L(1, \Pi_{\pi, v}, \mathrm{Sym}^2)}{\prod_{j=1}^2 \zeta_{F_v}(2j)} \mathcal{W}_{G, v}(\varphi_v)\label{mathcal W_G} \]

and $\mathcal {W}_{G, v}(\varphi _v)$ is defined by

\[ \mathcal{W}_{G, v}(\varphi_v) = \int_{U_G(F_v)}^{\rm st} \frac{(\pi_v^\circ(n)\varphi_v, \varphi_v)}{(\varphi_v, \varphi_v)} \,\psi_{U_G}^{-1}(n) \, dn \]

when $v$ is finite and by the Fourier transform when $v$ is archimedean.

6.2.2 Reduction to a local identity

Let us go back to the situation stated in the beginning of § 6.1.

First we note that the unramified computation in [Kud86] implies the following lemma.

Lemma 6.1 There exists a finite set $S_0$ of places of $F$ containing all archimedean places such that for a place $v\notin S_0$, we have

\[ L (1, \Pi_{\Sigma_v}, \mathrm{As}^+) = L(1, \pi_v, \mathrm{std} \otimes \chi_E) L(1, \Pi_{\pi,v}, \mathrm{Sym}^2)L(1, \chi_{E_v}). \]

Let us recall the following pull-back formula for the Whittaker period on $\Sigma = \theta _\psi (\pi ^\circ )$.

Proposition 6.1 [Mor14a, p. 40]

Let $f \in V_{\pi ^\circ }$ and $\phi \in \mathcal {S}(Z_+(\mathbb {A}))$. Then

(6.2.4) \begin{align} W^{\psi_U}(\theta(\phi; f)) &= \int_{N(\mathbb{A}) \backslash G^1(\mathbb{A})} (\omega_{\psi}(g_1, 1)\phi)( (x_{-2}, x_{-1}, 0, x_2)) W_{\psi_{U_G}}(\pi^\circ(g_1) f) \, dg_1. \end{align}

Suppose that $f=\otimes f_v$ and $\phi =\otimes \phi _v$. Then by an argument similar to that in obtaining [FM21, (2.27)], when $W_{\psi _{U_G}}(f) \ne 0$, we have

\[ W^{\psi_U}(\theta(\phi; f)) = C_{G^1} \cdot W_{\psi_{U_G}}(f) \cdot \prod_v \mathcal{L}_v^\circ(\phi_v, f_v), \]

where

\begin{align*} \mathcal{L}_v^\circ(\phi_v, f_v) &:= \int_{N(F_v) \backslash G^1(F_v)} (\omega_{\psi_v}(g_1, 1)\phi_v)((x_{-2}, x_{-1}, 0, x_2))\mathcal{W}_{G, v}^\circ(\pi_v^\circ(g_1)f_v) \, dg_{1,v} \end{align*}

when $\phi = \otimes _v \phi _v$ and $f = \otimes _v f_v$. We also define

\[ \mathcal{L}_v(\phi_v, f_v): = \frac{\prod_{j=1}^2 \xi_f(2j)}{L(1,\Pi_{\pi, v}, \mathrm{Sym}^2)} \frac{\mathcal{L}_v^\circ(\phi_v, f_v)}{ \mathcal{W}_{G, v}(f_v)}.\label{mathcal L} \]

Here the measures are taken as the following. Let $dg_v$ be the measure on $G^1(F_v)$ defined by the gauge form and $dn_v$ the measure on $N(F_v)$ defined in the manner stated in § 1.2. Then we take the measure $dg_{1,v}$ on $N(F_v) \backslash G^1(F_v)$ so that $dg_v=dn_v \, dg_{1,v}$.

Let $\Theta (\pi _v^\circ,\psi _v):=\operatorname {Hom}_{G(F_v)^+} (\Omega _{\psi _v},\bar {\pi }_v^\circ )$ where $\Omega _{\psi _v}$ is the extended local Weil representation of $G(F_v)^+ \times \mathrm {GSO}_{4,2}(F_v)$ realized on $\mathcal {S}(Z_+(F_v))$, the space of Schwartz–Bruhat functions on $Z_+(F_v)$. We recall that the action of $G(F_v)^+ \times \mathrm {GSO}_{4,2}(F_v)$ on $\mathcal {S}(Z_+(F_v))$ via $\Omega _{\psi _v}$ is defined as in the global case (see, e.g., [Mor14a, 2.2]). We also recall that for $\Sigma =\theta _\psi (\pi ^\circ )$, we have $\Sigma =\otimes _v \Sigma _v$ where $\Sigma _v=\theta _{\psi _v}(\pi _v^\circ )$ is the local theta lift of $\pi _v^\circ$.

Let

\[ \theta_v:\mathcal{S}(Z_+(F_v))\otimes V_{\pi_v^\circ}\to V_{\Sigma_v} \]

be a $G(F_v)^+ \times \mathrm {GSO}_{4,2}(F_v)$-equivariant linear map, which is unique up to a scalar multiplication. Since the global mapping

\[ \mathcal{S}(Z_+(\mathbb{A}))\otimes V_{\pi^\circ}\ni(\phi^\prime,f^\prime)\mapsto \theta_\psi(\phi^\prime; f^\prime)\in V_\Sigma \]

is $G(F_v)^+ \times \mathrm {GSO}_{4,2}(F_v)$-equivariant at any place $v$, by the uniqueness of $\theta _v$, we may adjust $\{\theta _v\}_v$ so that

\[ \theta_\psi (\phi^\prime; f^\prime)=\otimes_v \theta_v(\phi_v^\prime\otimes f_v^\prime)\quad\text{for $f^\prime=\otimes_v f_v^\prime\in V_{\pi^\circ}$, $\phi^\prime=\otimes_v\, \phi_v^\prime \in\mathcal{S}(Z_+(\mathbb{A}))$.} \]

Then as in [FM21, § 2.4], combining Theorem 6.3, the Rallis inner product formula (5.1.4), Lemmas 6.1 and 4.2 and Proposition 6.1, we see that a proof of Theorem 6.1 is reduced to a proof of the following local identity (6.2.5).

Proposition 6.2 Let $v$ be an arbitrary place of $F$. For a given $f_v \in V_{\pi ^\circ _v}^\infty$ satisfying $\mathcal {W}_{G_,v}(f_v) \ne 0$, there exists $\phi _v \in \mathcal {S}(Z_+(F_v))$ such that the local integral $\mathcal {L}_v(\phi _v, f_v)$ converges absolutely, $\mathcal {L}_v(\phi _v, f_v) \ne 0$ and the equality

(6.2.5)\begin{equation} \frac{Z_v(\phi_v, f_v, \pi_v) \cdot \mathcal{W}_v (\theta(\phi_v \otimes f_v))}{|\mathcal{L}_v(\phi_v, f_v)|^2} = \mathcal{W}_{G_, v}(f_v) \end{equation}

holds with respect to the specified local measures.

Let us define a hermitian inner product $\mathcal {B}_{\omega _v}$ on $\mathcal {S}(Z_+(F_v))$ by

\[ \mathcal{B}_{\omega_v}(\phi, \phi^\prime) = \int_{Z_+(F_v)} \phi(x)\overline{\phi^\prime}(x) \,{dx}\quad \text{for}\ \phi, \phi^\prime \in \mathcal{S}(Z_+(F_v)). \]

Here on $Z_+(F_v) \simeq (F_v)^{12}$, we take the product measure of the one on $F_v$. Then we consider the integral

(6.2.6) \begin{align} Z^{\flat}(f, f^\prime; \phi, \phi^\prime) &= \int_{G^1(F_v)} \langle \pi_v^\circ(g) f, f^\prime \rangle_v \mathcal{B}_{\omega_v}(\omega_{\psi}(g)\phi, \phi^\prime) \, dg \nonumber\\ &=\int_{G^1(F_v)} \int_{Z_+(F_v)} \langle \pi_v^\circ(g) f, f^\prime \rangle_v (\omega_{\psi_v}(g,1)\phi)(z) \overline{\phi^\prime(z)} \, {dx} \,dg \quad\text{for $f, f^\prime \in V_{\pi_v^\circ}$}. \end{align}

The integral (6.2.6) converges absolutely by Yamana [Yam14, Lemma 7.2]. As in Gan and Ichino [GI14, 16.5], we may define a $\mathrm {GSO}_{4,2}(F_v)$-invariant hermitian inner product $\mathcal {B}_{\Sigma _v} : V_{\Sigma _v} \times V_{\Sigma _v} \rightarrow \mathbb {C}$ by

\[ \mathcal{B}_\Sigma(\theta(\phi \otimes f), \theta(\phi^\prime \otimes f^\prime)) := Z^{\flat}(f, f^\prime; \phi, \phi^\prime). \]

Here we note that for $h \in \mathrm {SO}_{4,2}(F_v)$, we have

\[ \mathcal{B}_\Sigma(\Sigma(h)\theta(\phi \otimes f), \theta(\phi^\prime \otimes f^\prime)) =\mathcal{B}_\Sigma(\theta(\omega_{\psi}(1,h)\phi \otimes f), \theta(\phi^\prime \otimes f^\prime)). \]

As in the definition of $W_v$, we define

\[ \mathcal{W}^{\psi_U} (\tilde{\phi}_1, \tilde{\phi}_2):= \int_{U(F_v)}^{\rm st} \mathcal{B}_\Sigma(\Sigma(n)\tilde{\phi}_1, \tilde{\phi}_2) \psi_{U}(n)^{-1} \, dn \quad\text{for $\widetilde{\phi}_i \in \Sigma_v\ (i=1,2)$}.\label{mathcal W psi U} \]

Then by an argument similar to that in [FM21, 3.2–3.3], indeed word for word, Proposition 6.2 is reduced to the following another local identity, which is regarded as a local pull-back computation of the Whittaker periods with respect to the theta lift.

Proposition 6.3 For any $f, f^\prime \in V_{\pi _v^\circ }$ and any $\phi,\phi ^\prime \in C_c^\infty (Z_+(F_v))$, we have

(6.2.7)\begin{align} \mathcal{W}_{\psi_U}(\theta(\phi\otimes f), \theta(\phi^\prime\otimes f^\prime)) &= \int_{N(F_v)\backslash G^1(F_v)}\int_{N(F_v)\backslash G^1(F_v)} \nonumber\\ &\quad \times\mathcal{W}_{G,v}(\pi_v^\circ(g)f,\pi_v^\circ(g^\prime)f^\prime) (\omega_{\psi_v}(g,1)\phi)(x_0) \overline{(\omega_{\psi_v}(g^\prime,1)\phi^\prime)(x_0)}\, dg\, dg^\prime. \end{align}

Remark 6.1 Since $\{ g \cdot x_0 : g \in G^1(F_v) \}$ is locally closed in $Z_+(F_v)$, the mappings

\[ N(F_v)\backslash G^1(F_v) \ni g \mapsto \phi(g^{-1} \cdot x_0)\in\mathbb{C},\quad N(F_v)\backslash G^1(F_v) \ni g^\prime \mapsto \phi^\prime(g^{-1} \cdot x_0)\in \mathbb{C} \]

are compactly supported and, thus, the right-hand side of (6.2.7) converges absolutely for $\phi, \phi ^\prime \in C_c^\infty (Z_+(F_v))$.

6.2.3 Local pull-back computation

Here we shall prove Proposition 6.3 and, thus, complete our proof of Theorem 6.1.

Since we work over a fixed place $v$ of $F$, we shall suppress $v$ from the notation in this subsection, e.g. $F$ means $F_v$. Further, for any algebraic group $K$ over $F$, we denote its group of $F$-rational points $K(F)$ by $K$ for simplicity.

The case when $F$ is non-archimedean

Suppose that $F$ is non-archimedean. From the definition, the local Whittaker period is equal to

\[ \int_{U}^{\rm st} \int_{G^1} \int_{Z_+} (\omega_{\psi}(g, n) \phi)(x) \overline{\phi^\prime(x)} \langle \pi^\circ(g)f, f^\prime \rangle \psi_{U}(n)^{-1} \, {dx} \, dg \, dn. \]

Recall that we have defined subgroups $N_0, N_1, N_2$ and $\widetilde {U}$ of $U$ in (3.1.10), (3.1.11), (3.1.12) and (6.1.1), respectively. Then because of the absolute convergence of the integral (6.2.6), the above local integral can be written as

(6.2.8)\begin{align} & \int_{\widetilde{U}}^{\rm st} \int_{N}^{\rm st} \int_{N_1} \int_{N_0} \int_{Z_+} \int_{G^1} (\omega_{\psi}(g, u_0 u_1 u_2 \tilde{u}) \phi)(x) \overline{ \phi^\prime(x)} \nonumber\\ &\quad \times \langle \pi^\circ(g)f, f^\prime \rangle \psi_{U}(u_2 \tilde{u})^{-1} \, {dx} \, dg \, du_0\, du_1 \, du_2\, d\tilde{u}. \end{align}

Let us define $Z_{+, \circ } := \{ (a_1, a_2 ; 0,0) \in Z_+ : a_1\ {\rm and}\ a_2\ {\rm are\ linearly\ independent}\}$. Then since $Z_{+, \circ } \oplus (X_+ \otimes Y_0)$ is open and dense in $Z_+$, we have

\[ \int_{Z_+} \Phi(z) \, dz = \int_{Z_{+, \circ}} \int_{X_+ \otimes Y_0} \Phi(z_1 +z_2) \, dz_2 \, dz_1 \]

for any $\Phi \in L^1(Z_+)$. We consider a map $p : Z_{+, \circ } \rightarrow F$ defined by $p((a_1, a_2; 0, 0)) = \langle a_1, a_2\rangle$. This is clearly surjective. For each $t \in F$, we fix $x_t \in Z_{+, \circ }$ such that $p(x_t) = t$. Then by Witt's theorem, the fiber $p^{-1}(x_t)$ of $x_t :=(a_1^t, a_2^t; 0, 0)$ is given by

\[ p^{-1}(x_t) = \{\gamma \cdot x_t :=(\gamma a_1^t, \gamma a_2^t: 0, 0) : \gamma \in G^1\}. \]

We may identify this space with $G^1 / R_t$ as a $G^1$-homogeneous space. Here $R_t$ denotes the stabilizer of $x_t$ in $G^1$. From this observation, the following lemma readily follows (cf. [FM21, Lemma 3]).

Lemma 6.2 For each $x_t \in Z_{+, \circ }$, there exists a Haar measure $dr_t$ on $R_t$ such that

\[ \int_{Z_+}\Phi(z) \, dz = \int_{F} \int_{R_t \backslash G^1} \int_{X_+ \otimes Y_0} \Phi(g^{-1} \cdot x_t+z) \, dz \, dg_t \, dt. \]

Here $dg_t$ denotes the quotient measure $dr_t \backslash dg$ on $R_t \backslash G^1$.

Further, we note that the following lemma, which is proved by an argument similar to that for [Liu16, Lemma 3.20] (cf. [FM21, Lemma 3]).

Lemma 6.3 For $\phi _1,\phi _2\in C_c^\infty \big (Z_+\big )$ and $f_1,f_2\in V_{\pi ^\circ }$, let

\[ \mathcal{G}_{\phi_1,\phi_2,f_1,f_2}(t)=\int_{G^1} \int_{R_t \backslash G^1}\phi_1((g g^\prime)^{-1} \cdot x_t) \phi_2(g^{-1} \cdot x_t)\langle\pi^\circ(g^\prime)f_1,f_2\rangle\, dg\, dg^\prime \]

for $t \in F$. Then the integral is absolutely convergent and is locally constant.

Remark 6.2 When $F$ is archimedean, by an argument similar to that for [Liu16, Proposition 3.22], we see that this integral is absolutely convergent and is a continuous function on $F$ not only for $C_c^\infty (Z_+)$ but also for $\mathcal {S}(Z_+)$.

By Lemma 6.2, the integral (6.2.8) can be written as

\begin{align*} & \int_{N_0} \int_{F} \int_{R_t \backslash G^1} \int_{X_+ \otimes Y_0} \int_{G^1} (\omega_{\psi}(g, u_0 h) \phi)(\gamma^{-1} \cdot x_t+z) \overline{\phi^\prime(\gamma^{-1} \cdot x_t+z)} \\ &\quad \times\langle \pi^\circ(g)f, f^\prime \rangle \, dg \, dz \, d\gamma_t \, dt \, du_0. \end{align*}

Moreover, by the computation in [Mor14a, § 3.1], we have

\[ (\omega_{\psi}(g, u_0(x) h) \phi)(\gamma^{-1} \cdot x_t+z) = \psi(-xt) \phi(\gamma^{-1} \cdot x_t+z). \]

Then because of Lemma 6.3, we may apply the Fourier inversion with respect to $x$ and $t$, and thus the above integral is equal to

(6.2.9)\begin{align} & \int_{R_0 \backslash G^1} \int_{X_+ \otimes Y_0} \int_{G^1} (\omega_{\psi}(g, h) \phi)(\gamma^{-1} \cdot x_0+z) \overline{ \phi^\prime(\gamma^{-1} \cdot x_0+z)} \nonumber\\ &\qquad \times \langle \pi^\circ(g)f, f^\prime \rangle\, dg \, dz \, d\gamma_0 \, dt \, du_0 \nonumber\\ &\quad = \int_{R_0 \backslash G^1} \int_{X_+ \otimes Y_0} \int_{G^1} (\omega_{\psi}(\gamma g, h) \phi)(x_0+z) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+z)} \nonumber\\ &\qquad \times \langle \pi^\circ(g)f, f^\prime \rangle\, dg \, dz \, d\gamma_0. \end{align}

The support of $\phi ^\prime (\gamma ^{-1} \cdot x_0+z)$ as a function of $X_+ \otimes Y_0$ is compact since $\phi ^\prime \in C_c^\infty (Z_+)$. Therefore this integral converges absolutely and is equal to

\begin{align*} & \int_{X_+ \otimes Y_0} \int_{R_0 \backslash G^1} \int_{G^1} (\omega_{\psi}(\gamma g, h)\phi)(x_0+z) (\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(x_0+z)} \langle \pi^\circ(g)f, f^\prime \rangle \, dg \, d\gamma_0 \, dz. \end{align*}

Now, let us take $(x_{-2}, x_{-1}: 0, 0)$ as $x_0$. Then we have

\[ R_0 = N. \]

Let us define a map $q : X_+ \otimes Y_0 \rightarrow \mathrm {Mat}_{2 \times 2}$ by

\[ q(b_1 \otimes e_1+b_2 \otimes e_2) = \begin{pmatrix} \langle x_{-2}, b_1 \rangle & \langle x_{-2}, b_2 \rangle \\ \langle x_{-1}, b_1 \rangle & \langle x_{-1}, b_2 \rangle \end{pmatrix} \]

with $b_i \in X_+$. Clearly this map is bijective. Hence, there exists a measure $dT$ on $\mathrm {Mat}_{2 \times 2}$ such that we have

\[ \int_{X_+ \otimes Y_0} \Phi(x_{-2}, x_{-1}: z) \, dz = \int_{\mathrm{Mat}_{2 \times 2}} \Phi(x_{-2}, x_{-1}: x_T) \, dT \]

with $x_T = q^{-1}(T)$. Here we note that the measure $dz$ on $X_+ \otimes Y_0$ is taken to be the Tamagawa measure and hence we have the Fourier inversion

\[ \int_{\mathrm{Mat}_{2 \times 2}} \int_{\mathrm{Mat}_{2 \times 2}} \Phi(T) \psi (\mathrm{tr} (T S_0 T^\prime)) \, dT \, dT^\prime = \Phi(0) \]

with the above Haar measures $dT, dT^\prime$ on $\mathrm {Mat}_{2 \times 2}$ if the integral converges. Thus, we have

\begin{align*} & \int_{N_2}^{\rm st} \int_{N_1}\int_{X_+ \otimes Y_0} \int_{N \backslash G^1} \int_{G^1} (\omega_{\psi}(\gamma g, u_1 u_2 h)\phi)(x_0+z) (\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(x_0+z)} \\ &\qquad \times\langle \pi^\circ(g)f, f^\prime \rangle \, dg \, d\gamma_0 \, dz \, du_1 \, du_2\\ &\quad =\int_{N}^{\rm st} \int_{N_1}\int_{\mathrm{Mat}_{2 \times 2}} \int_{N \backslash G^1} \int_{G^1} (\omega_{\psi}(\gamma g, u_1 u_2 h)\phi)(x_0+x_T) (\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(x_0+x_T)} \\ &\qquad \times \langle \pi^\circ(g)f, f^\prime \rangle\, dg \, d\gamma_0 \, dT \, du_1 \, du_2. \end{align*}

Moreover, similarly to the global computation in [Mor14a, § 3.1], we may write this integral as

(6.2.10)\begin{align} & \int_{N_2}^{\rm st} \int_{N_1} \int_{\mathrm{Mat}_{2 \times 2}} \int_{N \backslash G^1} \int_{G^1} \psi \bigg(\mathrm{tr} \bigg(\begin{pmatrix}s_1 & s_2\\ t_1 & t_2 \end{pmatrix}S_0 (x_T-x_{T_0})\bigg)\bigg) \nonumber\\ &\quad \times (\omega_{\psi}(\gamma g, h)\phi)(x_0+x_T) (\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(x_0+x_T)} \langle \pi^\circ(g)f, f^\prime \rangle \, dg \, d\gamma_0 \, dT \, du_1 \, du_2 , \end{align}

where we let $u_1 = u_1(s_1, t_1)$ and $u_2 = u_2(s_2, t_2)$, and we put $T_0 = \big (\begin{smallmatrix}0 & 0\\0 & 1 \end{smallmatrix}\big )$. By an argument similar to the proof to show (6.2.9), we may apply the Fourier inversion to this integral, and we see that this is equal to

\[ \int_{N_H \backslash G^1} \int_{G^1}(\omega_{\psi}(\gamma g, h) \phi)(x_0+x_{T_0}) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+x_{T_0})} \langle \pi^\circ(g)f, f^\prime \rangle \, dg \, d\gamma_0. \]

Now we note that from the argument to obtain (6.2.9), this integral converges absolutely. Then by telescoping the $G^1$-integration, we obtain

\begin{align*} & \int_{N \backslash G^1} \int_{N \backslash G^1} \int_{N} (\omega_{\psi}( r g, h) \phi)(x_0+x_{T_0}) (\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(x_0+x_{T_0})} \\ &\quad \times \langle \pi^\circ(r g)f, \pi^\circ(\gamma)f^\prime \rangle\, dr \, dg \, d\gamma_0. \end{align*}

Put $z_0 = x_0+x_{T_0} = (x_{-2}, x_{-1}, 0, x_2)$. Recall that from the computation in [Mor14a, § 3.1], we have

(6.2.11)\begin{equation} \omega_{\psi}(v(A)g, \tilde{u}(b) h) \phi(z_0) = \psi(-d a_{22})\omega_{\psi}(g, \tilde{u}(b) h) \phi(z_0) \end{equation}

when we let $A =\big (\begin{smallmatrix}a_{11} & a_{12}\\a_{21} & a_{22} \end{smallmatrix}\big )$, and we have

(6.2.12)\begin{equation} z_0(1, \tilde{u}(b)) = z_0(w(b), 1). \end{equation}

Therefore, $\mathcal {W}_{\psi _U}(\theta (\phi \otimes f), \theta (\phi ^\prime \otimes f^\prime ))$ is equal to

\begin{align*} & \int_{F}^{\rm st}\int_{N \backslash G^1} \int_{N \backslash G^1} \int_{N} \psi(-b) (\omega_{\psi}(r g, \tilde{u}(b)) \phi)(z_0) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(z_0)} \\ &\qquad \times \langle \pi^\circ(rg)f, \pi^\circ(\gamma)f^\prime \rangle\, dr \, dg \, d\gamma_0 \, db \\ &\quad =\int_{F}^{\rm st}\int_{N \backslash G^1} \int_{N \backslash G^1} \int_{\mathrm{Sym}^2} \psi(-b-da_{22})(\omega_{\psi}(w(b)g, 1)\phi)(z_0)\\ &\qquad \times(\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(z_0)} \langle \pi^\circ(v(A)g)f, \pi^\circ(\gamma)f^\prime \rangle \, dA \, dg \, d\gamma_0 \, db. \end{align*}

By an argument similar to that in [FM21] showing that [FM21, (3.30)] is equal to $\alpha (\pi (g) \phi, \pi (h)\phi ^\prime )$ there, indeed word for word, we see that this integral is equal to

\begin{align*} & \int_{N \backslash G^1} \int_{N \backslash G^1} \int_{U_G}^{\rm st} \psi_{U_{G}}^{-1}(n) (\omega_{\psi}(g, 1)\phi)(z_0)(\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(z_0)} \langle \pi^\circ(ng)f, \pi^\circ(\gamma)f^\prime \rangle\, dn \, dg \, d\gamma_0. \end{align*}

Thus, Proposition 6.3 in the non-archimedean case is proved.

The case when $F$ is archimedean

Suppose that $F$ is archimedean. Recall that

\[ \mathcal{W}^{\psi_U}(\tilde{\phi}_1, \tilde{\phi}_2)= \widehat{\mathcal{W}_{\tilde{\phi}_1, \tilde{\phi}_2}}(\psi_U) \quad \text{for $\tilde{\phi}_i \in \Sigma^\infty\ (i=1,2)$}, \]

where we set

\[ \mathcal{W}_{\tilde{\phi}_1, \tilde{\phi}_2}(n) = \int_{U_{-\infty}} \mathcal{B}_{\Sigma}(\Sigma(n u)\tilde{\phi}_1, \tilde{\phi}_2) \psi_{U}^{-1}(n u) \, du \quad\text{for $n\in U$}, \]

which converges absolutely and gives a tempered distribution on $U / U_{-\infty }$ by [Liu16, Corollary 3.13]. Let us define $U^\prime = N_0 N_1 N_2$. Then $U^\prime _{-\infty } = U_{-\infty }$. Moreover, for any $\tilde {u} \in \widetilde {U}$ and $u^\prime \in U^\prime$, we have $\tilde {u} u^\prime \tilde {u}^{-1} (u^\prime )^{-1} \in U^\prime _{-\infty }$ and we obtain $\mathcal {W}_{\tilde {\phi }_1, \tilde {\phi }_2}(\tilde {u} u^\prime ) = \mathcal {W}_{\tilde {\phi }_1, \tilde {\phi }_2}(u^\prime \tilde {u})$. Hence, we may regard it as a tempered distribution on $\tilde {U} \times (U^\prime / U_{-\infty }^\prime )$. Then for a tempered distribution $I$ on $\tilde {U} \times (U^\prime / U_{-\infty }^\prime )$, we define partial Fourier transforms $\widehat {I^j}$ of $I$ for $j=1,2$ by

\[ \langle I, \widehat{f_1} \otimes f_2 \rangle = \langle \widehat{I^1}, f_1 \otimes f_2 \rangle\quad \text{and}\quad \langle I, f_1 \otimes \widehat{f_2} \rangle = \langle \widehat{I^2}, f_1 \otimes f_2 \rangle, \]

where $f_1 \in \mathcal {S} (\tilde {U})$ and $f_2 \in \mathcal {S} (U^\prime / U_{-\infty }^\prime )$, respectively. Then we have

\[ \widehat{\widehat{I}^2}^1(\psi_U) = \widehat{\widehat{I}^1}^2(\psi_U)= \widehat{I}(\psi_U). \]

From the definition of $\mathcal {B}_\Sigma$, we have

\begin{align*} \mathcal{W}_{\theta(\phi \otimes f), \theta(\phi^\prime \otimes f^\prime)}(n) &= \int_{U_{-\infty}} \int_{G^1} \int_{Z_+} (\omega_{\psi}(g, nu) \phi)(x) \overline{\phi^\prime(x)}\\ &\quad \times \langle \pi^\circ(g)f, f^\prime \rangle \psi_{U}^{-1}(n u) \, {dx} \, dg \, du\\ &=\int_{U_{-\infty} / N_0} \int_{N_0} \int_{G^1} \int_{Z_+} (\omega_{\psi}(g, nu_0u) \phi)(x) \overline{ \phi^\prime(x)}\\ &\quad \times \langle \pi^\circ(g)f, f^\prime \rangle \psi_{U}^{-1}(n u) \, {dx} \, dg \, du_0 \, du, \end{align*}

for $\phi, \phi ^\prime \in \mathcal {S}(Z_+)$ and $f, f^\prime \in V_{\pi ^\circ }^\infty$. Clearly, Lemma 6.2 holds in the archimedean case also. Then as in (6.2.9), because of Remark 6.2 and the Fourier inversion, the above integral is equal to

\begin{align*} & \int_{U_{-\infty} / N_0} \int_{N \backslash G^1} \int_{X_+ \otimes Y_0} \int_{G^1} (\omega_{\psi}(\gamma g, nu) \phi)(x_0+z) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+z)} \\ &\qquad \times\langle \pi^\circ(g)f, f^\prime \rangle \, dg \, dz \, d\gamma_0 \, du. \end{align*}

As (6.2.9), this integral converges absolutely. Let us denote this integral by $J_{\phi, \phi ^\prime, f, f^\prime }(n)$. Then, from the definition,

\[ \widehat{J_{\phi, \phi^\prime, f, f^\prime}} = \widehat{\mathcal{W}_{\theta(\phi \otimes f), \theta(\phi^\prime \otimes f^\prime)}}. \]

Again, from the definition, for $\varphi \in \mathcal {S}(U^\prime / U^\prime _{-\infty })$, we have

\begin{align*} (\widehat{J_{\phi, \phi^\prime, f, f^\prime}}^2, \psi_U \cdot \varphi) = (J_{\phi, \phi^\prime, f, f^\prime}, \widehat{\psi_U \cdot \varphi}) &= \int_{U^\prime / U^\prime_{-\infty}} \int_{U_{-\infty}^\prime / N_0} \int_{N \backslash G^1} \int_{X_+ \otimes Y_0} \int_{G^1} \\ &\quad \times (\omega_{\psi}(\gamma g, nu) \phi)(x_0+z) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+z)} \\ &\quad \times\langle \pi^\circ(g)f, f^\prime \rangle \widehat{\varphi}(n) \psi_U^{-1}(n)\, dg \, dz \, d\gamma_0 \, du \, dn. \end{align*}

By a computation similar to that used to obtain (6.2.10), this integral is equal to

\begin{align*} & \int_{N_1} \int_{N_2} \int_{N \backslash G^1} \int_{X_+ \otimes Y_0} \int_{G^1}(\omega_{\psi}(\gamma g, u_1 u_2u) \phi)(x_0+z) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+z)} \\ &\qquad \times \langle \pi^\circ(g)f, f^\prime \rangle \widehat{\varphi}(u_1 u_2) \psi_U^{-1}(u_1 u_2)\, dg \, dz \, d\gamma_0 \, du \, du_1 \, du_2\\ &\quad =\int_{N_1} \int_{N_2}\int_{\mathrm{Mat}_{2 \times 2}} \int_{N \backslash G^1} \int_{G^1} \psi \bigg(\mathrm{tr} \bigg(\begin{pmatrix}s_1 & s_2\\ t_1 & t_2 \end{pmatrix}S_0 (x_T-x_{T_0}) \bigg) \bigg) \\ &\qquad \times (\omega_{\psi}(\gamma g, h)\phi)(x_0+x_T) (\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(x_0+x_T)} \\ &\qquad \times\langle \pi^\circ(g)f, f^\prime \rangle \widehat{\varphi}(u_1 u_2)\, dg \, d\gamma_0 \, dT \, du_1 \, du_2. \end{align*}

As above, we may apply the Fourier inversion and, thus, this is equal to

\begin{align*} & \widehat{\varphi}(1) \cdot \int_{N \backslash G^1} \int_{G^1} (\omega_{\psi}(\gamma g,1) \phi)(x_0+x_{T_0}) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+x_{T_0})}\langle \pi^\circ(g)f, f^\prime \rangle \, dg \, d\gamma_0. \end{align*}

Hence,

\begin{align*} \widehat{J_{\phi, \phi^\prime, f, f^\prime}}^2(\psi_U) &= \int_{N \backslash G^1} \int_{G^1} (\omega_{\psi}(\gamma g,1) \phi)(x_0+x_{T_0}) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+x_{T_0})} \langle \pi^\circ(g)f, f^\prime \rangle \, dg \, d\gamma_0. \end{align*}

Here, we note that by Remark 6.2, this integral converges absolutely. Then this identity shows that we have

(6.2.13)\begin{align} \widehat{\widehat{J_{\phi, \phi^\prime, f, f^\prime}}^2}^1(\varphi) &= \int_{\tilde{U}} \int_{N \backslash G^1} \int_{G^1} (\omega_{\psi}(\gamma g,b) \phi)(x_0+x_{T_0}) (\overline{\omega_{\psi}(\gamma, 1) \phi^\prime)(x_0+x_{T_0})} \nonumber\\ &\quad \times \langle \pi^\circ(g)f, f^\prime \rangle \varphi(b)\, dg \, d\gamma_0 \, db \end{align}

for $\varphi \in \mathcal {S}(\tilde {U})$. As in the non-archimedean case, by (6.2.11) and (6.2.12), we may easily show that this is equal to

\begin{align*} & \int_{N \backslash G^1} \int_{N \backslash G^1} \int_{N} \int_{F} \psi_{U_{G}}^{-1}(v(x) n) (\omega_{\psi}(g, 1)\phi)(z_0)(\overline{\omega_{\psi}(\gamma, 1)\phi^\prime)(z_0)} \nonumber\\ &\quad \times \langle \pi^\circ(v(x)ng)f, \pi^\circ(\gamma)f^\prime \rangle \varphi(\tilde{u}(x)) \,{dx}\, dn \, dg \, d\gamma_0 \end{align*}

since the integral in (6.2.13) converges absolutely. Thus, Proposition 6.3 is proved in the archimedean case also.

7. Proof of Theorem 1.2

In this section, we complete our proof of Theorem 1.2. Let $(\pi, V_\pi )$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with a trivial central character. Throughout this section, we suppose that $\pi$ is neither of type I-A nor type I-B. When $\pi$ is one of these types, our theorem is already proved in [Cor17, Theorem 7.5].

The case when $B_{\xi, \Lambda,\psi } \not \equiv 0$ on $V_\pi$ is treated in § 7.1 and the case when $B_{\xi, \Lambda, \psi } \equiv 0$ on $V_\pi$ is treated in § 7.2, respectively.

7.1 Proof of Theorem 1.2 when $B_{\xi, \Lambda,\psi } \not \equiv 0$

7.1.1 Reduction to a local identity

Suppose that $B_{\xi, \Lambda,\psi } \not \equiv 0$ on $V_\pi$. Let $(\sigma, V_\sigma )$ denote the theta lift of $\pi$ to $\mathrm {GSU}_{3,D}(\mathbb {A})$, which is an irreducible cuspidal automorphic representation. As in the proof of Theorem 6.1, our theorem may be reduced to a certain local identity. Let us set some notation to explain our local identity.

As in §§ 5.1 and 5.2, we fix the Petersson inner product $(\,{,}\,)$ on $V_{\pi }$ and the local hermitian pairing $(\,{,}\,)_v$ on $\pi _v$. As in (3.2.3), we define the maximal isotropic subspaces $Z_{D, \pm }$. Let

\[ \theta_{D, v}:\mathcal{S}\big(Z_{D, +}(F_v)\big)\otimes V_{\pi_v}\to V_{\sigma_v} \]

be the $G_D(F_v)^+ \times \mathrm {GSU}_{3, D}(F_v)$-equivariant linear map, which is unique up to multiplication by a scalar. As in § 6.1, let us adjust $\{\theta _{D, v}\}_v$ so that

\[ \theta_{D, \psi}(\phi^\prime; f^\prime)=\otimes_v \theta_{D, v}(\phi_v^\prime\otimes f_v^\prime) \]

for $f^\prime =\otimes _v f_v^\prime \in V_{\pi }$ and $\phi ^\prime =\otimes _v \phi _v^\prime \in \mathcal {S}(Z_{D, +}(\mathbb {A}))$. Let us choose $X \in D^\times (F)$ so that $S_{X} =\xi$. Then by Proposition 3.2, we have

(7.1.1)\begin{equation} \mathcal{B}_{X, \Lambda^{-1}}(\theta(f: \phi))=B_{\xi, \Lambda}(f) \cdot \prod_v \mathcal{K}_v(f_v; \phi_v), \end{equation}

where $f = \otimes f_v \in V_{\pi _D}$ and $\phi = \otimes \phi _v \in \mathcal {S}(Z_{D, +}(\mathbb {A}))$, and we define

\[ \mathcal{K}_v(f_v ;\phi_v) = \int_{N_D(F_v) \backslash G_D^1(F_v)} \alpha_{\Lambda_v, \psi_{\xi, v}}(\pi_v(g)f_v) \phi_v(g^{-1} \cdot v_{D, X})\,dg. \]

Here, we take the measure $dh_{v}$ on $G_D^1(F_v)$ defined by the gauge form, the measure $dn_v$ on $N_{G_D}(F_v)$ defined in § 1.2 under the identification $D(F_v) \simeq F_v^4$ and the measure $dg_{1,v}$ on $N_{G_D}(F_v) \backslash G_D^1(F_v)$ such that $dh_v=dn_v \, dg_{1,v}$.

Then by combining the explicit formula of the Bessel periods on $\sigma$ given in Theorem 6.2, the Rallis inner product formulas (5.1.4) and Proposition 5.2, Lemmas 6.1 and 4.2, and the above pull-back formula (7.1.1), we see that Theorem 1.2 is reduced to the following local identity.

Proposition 7.1 Let $v$ be an arbitrary place of $F$. For a given $f_v \!\in\! V_{\pi _v}$ satisfying $\alpha _{\xi, \Lambda, v}(f_v)\ne 0$, there exists $\phi _v \in \mathcal {S}(Z_{D, +}(F_v))$ such that the local integral $\mathcal {K}_v(f_v;\phi _v)$ converges absolutely, $\mathcal {K}_v(f_v;\phi _v) \ne 0$ and the equality

\[ \frac{Z_v(\phi_v, f_v, \pi_v) \alpha_{\Lambda_v^{-1}, \psi_{X, v}} (\theta(\phi_v \otimes f_v)}{|\mathcal{K}_v(f_v;\phi_v)|^2} = \frac{\alpha_{\Lambda_v, \psi_{\xi, v}}(f_v)}{(f_v, f_v)_v} \]

holds.

Remark 7.1 In Corollary 7.1, the existence of $f_v$ with $\alpha _{\Lambda _v, \psi _{\xi, v}}(f_v)\ne 0$ is shown.

Let us define hermitian inner product on $\mathcal {S}(Z_{D, +}(F_v))$ by

\[ \mathcal{B}_{\omega_v, D}(\phi, \phi^\prime) = \int_{Z_{D, +}(F_v)} \phi(x) \overline{\phi^\prime}(x) \,{dx}\quad \text{for}\ \phi, \phi^\prime \in \mathcal{S}(Z_{D, +}(F_v)). \]

Then we consider the integral

\[ Z^{\bullet}(f, f^\prime; \phi, \phi^\prime) = \int_{G^1(F_v)} \langle \pi_v(g) f, f^\prime \rangle_v \mathcal{B}_{\omega_v} (\omega_{\psi}(g)\phi, \phi^\prime) \, dg \]

for $f, f^\prime \in \pi _v$ and $\phi, \phi ^\prime \in \mathcal {S}(Z_{D, +}(F_v))$. As in § 6.2, this converges absolutely and gives a $\mathrm {GSU}_{3,D}(F_v)$-invariant hermitian inner product

\[ \mathcal{B}_{\sigma_v} : V_{\sigma_v} \times V_{\sigma_v} \rightarrow \mathbb{C} \]

by

\[ \mathcal{B}_{\sigma_v}(\theta(\phi \otimes f), \theta(\phi^\prime \otimes f^\prime)) := Z^{\bullet}(f, f^\prime; \phi, \phi^\prime). \]

By the Rallis inner product formula (5.1.4) and Proposition 5.2, at any place $v$, there exist $f_v, f_v^\prime, \phi, \phi ^\prime$ such that $Z^{\bullet }(f, f^\prime ; \phi, \phi ^\prime ) \ne 0$ since $\theta _{\psi, D}(\pi ) \ne 0$. Thus, $\mathcal {B}_{\sigma _v} \not \equiv 0$.

For $\widetilde {\phi }_i \in \sigma _v$, we define

\[ \mathcal{A}(\tilde{\phi}_1, \tilde{\phi}_2):= \int_{N_{3, D}(F_v)}^{\rm st} \int_{M_{X}(F_v)} \mathcal{B}_{\sigma_v}(\sigma_v(nt)\tilde{\phi}_1, \tilde{\phi}_2) \Lambda_{D,v}(t) \psi_{X, D, v}(n)^{-1} \, dt \, dn. \]

Here, at an archimedean place $v$, a stable integration means the Fourier transform as in the definition of $\alpha _{\chi, \psi _N}$. Then by an argument similar to that in [FM21, 3.2–3.3], we may reduce Proposition 7.1 to the following identity.

Proposition 7.2 For any $f, f^\prime \in V_{\pi _v}$ and any $\phi,\phi ^\prime \in C_c^\infty (Z_{D, +}(F_v))$, we have

(7.1.2) \begin{align} \mathcal{A}(\theta(\phi\otimes f),\theta(\phi^\prime\otimes f^\prime)) &=\int_{N_D(F_v)\backslash G_D^1(F_v)}\int_{N_D(F_v)\backslash G_D^1(F_v)} \alpha_{\Lambda_v, \psi_{\xi, v}}(\pi_v(h)f,\pi_v(h^\prime)f^\prime) \nonumber\\ &\quad \times (\omega_{\psi_v}(h,1)\phi)(x_0) \overline{(\omega_{\psi_v}(h^\prime,1)\phi^\prime)(x_0)}\, dh\, dh^\prime. \end{align}

Before proceeding to a proof of this proposition, we give some corollaries of this identity.

Corollary 7.1 For an arbitrary place $v$ of $F$, we have $\alpha _{\Lambda _v, \psi _{\xi, v}} \not \equiv 0$ on $\pi _v$.

Proof. Since $\mathcal {B}_{\sigma _v} \not \equiv 0$, (7.1.2) implies that $\alpha _{\Lambda _v, \psi _{\xi, v}} \not \equiv 0$ on $\pi _v$ if and only if $\alpha _{\Lambda _v^{-1}, \psi _{X, v}} \not \equiv 0$ on $\sigma _v$. Moreover, by [FM21, Corollary 5.1], $\alpha _{\Lambda _v^{-1}, \psi _{X, v}} \not \equiv 0$ on $\sigma _v$ since the theta lift of $\sigma _v$ to $\mathrm {GU}_{2,2}(F_v)$ is generic. Thus, our claim follows.

As another corollary, a non-vanishing of local theta lifts follows from a non-vanishing of local periods.

Corollary 7.2 Let $k$ be a local field of characteristic zero and $\mathcal {D}$ be a quaternion algebra over $k$. Let $\tau$ be an irreducible admissible tempered representation of $G_{\mathcal {D}}$ with a trivial central character. Let $S_{\mathcal {D}} \in \mathcal {D}^1$ and $\chi$ be a character of $T_{\mathcal {D}, S_{\mathcal {D}}}$. Suppose that $\alpha _{\chi, \psi _{S_{\mathcal {D}}}} \not \equiv 0$ on $\tau$. Then $\mathcal {A} \not \equiv 0$ on $\theta _{\psi, \mathcal {D}}(\tau ) \times \theta _{\psi, \mathcal {D}}(\tau )$. In particular, $\theta _{\psi, \mathcal {D}}(\tau ) \ne 0$ and $Z^\bullet (\phi, \phi ^\prime, f, f^\prime ) \ne 0$ for some $f, f^\prime \in \tau$ and $\phi, \phi ^\prime \in \mathcal {S}(Z_{\mathcal {D}, +})$.

Remark 7.2 By [Yam14, Lemma 8.6, Remark 8.4 (1)], we know that the existence of such $f, f^\prime, \phi, \phi ^\prime$ is equivalent to the non-vanishing of the theta lift of $\tau$ to $\mathrm {GSU}_{3,\mathcal {D}}$ when $k \ne \mathbb {R}$. Though the equivalence is not clear when $k=\mathbb {R}$, we shall use Corollary 7.2 to show that the local non-vanishing of the theta lifts implies the global non-vanishing of the theta lifts in § 7.2.

Proof. By our assumption, the right-hand side of (7.1.2) is not zero for some $f, f^\prime, \phi, \phi ^\prime$ when $F_v \ne \mathbb {R}$. Hence, the left-hand side is not zero and, in particular, $Z^\bullet (\phi, \phi ^\prime, f, f^\prime ) \ne 0$.

7.1.2 Local pull-back computation

Here we shall prove the identity (7.1.2) and, thus, we complete our proof of Theorem 1.2 when $B_{\xi, \psi, \Lambda } \not \equiv 0$. Here we give a proof of (7.1.2) only in the non-archimedean case since the archimedean case is similarly proved as in the proof of Proposition 6.3. Our proof is a local analogue of the proof of Propositions 3.1 and 3.2. Moreover, we will consider only the case when $D$ is split since the proof is similar and indeed is easier in the non-split case as in the global computation. Since the argument in this subsection is purely local, in order to simplify the notation, we omit subscripts $v$ and we simply write $K(F)$ by $K$ for any algebraic group $K$ defined over $F=F_v$.

From the definition, we may write the left-hand side of (7.1.2) as

\[ \int_{N_{4,2}}^{\rm st} \int_{M_{X}} \int_{G^1} \int_{Z_+} \langle \pi(g)f, f^\prime \rangle (\omega_{\psi}(g, nt) \phi)(x) \overline{\phi^\prime(x)}\Lambda(t) \psi_{X}(n)^{-1} \, {dx} \, dg \, dt \, dn, \]

where $X$ is chosen so that $S_X =S$. Further as in (3.1.13), this is equal to

\begin{align*} & \int_{F}^{\rm st} \int_{F^2}^{\rm st} \int_{F^2}^{\rm st} \int_{M_{X}} \int_{G^1} \int_{Z_+} (\omega_{\psi}(g, u_0(s) u_1(s_1, t_1) u_2(s_2, t_2) t) \phi)(x) \overline{\phi^\prime(x)} \\ &\quad \times \langle \pi(g)f, f^\prime \rangle \Lambda(t) \psi(x_{21}s_1+x_{22}t_1+x_{11}s_2+x_{12}t_2)^{-1} \, {dx} \, dg \, dt \, ds_2 \,dt_2 \,ds_1 \, dt_1 \,ds \end{align*}

when we let $X = \big (\begin{smallmatrix} x_{11} & x_{12}\\ x_{21} & x_{22}\end{smallmatrix}\big )$. For each $r \in F$, we may take $A_r =(a_1^r, a_2^r, 0, 0) \in Z_+$ such that $a_1^r,a_2^r$ are linearly independent and $\langle a_1^r, a_2^r \rangle =r$. Let us denote by $Q_r$ the stabilizer of $x_r$ in $G^1$. Then as in the proof of Proposition 6.3, for each $r \in F$, there is a Haar measure $dq_r$ of $Q_r$ such that

\[ \int_{Z_+} \Phi(x) \, {dx} = \int_{F} \int_{Q_r \backslash G^1} \int_{X_+^2} \Phi(h^{-1} \cdot A_r+b) \, db \,dh_r \,dr \]

with $dh_r=dq_r \backslash dh$, provided that the both sides converge. Then applying the Fourier inversion, because of (3.1.15), our integral becomes

\begin{align*} & \int_{F^2}^{\rm st} \int_{F^2}^{\rm st} \int_{M_{X} }\int_{G^1}\int_{Q_0 \backslash G^1} \int_{X_+^2} \langle \pi(g)f, f^\prime \rangle\Lambda(t) \psi(x_{21}s_1+x_{22}t_1+x_{11}s_2+x_{12}t_2)^{-1} \\ &\quad \times(\omega_{\psi}(hg, u_1(s_1, t_1) u_2(s_2, t_2) t) \phi)(A_0+b) \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+b)} \\ &\quad \times db \, dh \, {dx} \, dg \, dt \, ds_2 \,dt_2 \,ds_1 \, dt_1 \end{align*}

with $A_0=(x_{-2}, x_{-1}, 0, 0)$. This is verified by an argument similar to that for [Liu16, Lemma 3.20]. We note that $Q_0 = N$ from the definition. Moreover, as in [Liu16, Lemma 3.19], the inner integral $\int _{M_{X}} \int _{G^1}\int _{Q_0 \backslash G^1} \int _{X_+^2}$ converges absolutely and, thus, this is equal to

\begin{align*} & \int_{F^2}^{\rm st} \int_{F^2}^{\rm st} \int_{Q_0 \backslash G^1} \int_{G^1} \int_{M_{X}} \int_{X_+^2} \langle \pi(g)f, f^\prime \rangle \Lambda(t) \psi(x_{21}s_1+x_{22}t_1+x_{11}s_2+x_{12}t_2)^{-1} \\ &\quad \times(\omega_{\psi}(hg, u_1(s_1, t_1) u_2(s_2, t_2) t) \phi)(A_0+b) \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+b)} \\ &\quad \times db \, dh \, {dx} \, dg \, dt \, ds_2 \,dt_2 \,ds_1 \, dt_1 . \end{align*}

From the proof of Lemma 3.2, this integral is equal to

(7.1.3)\begin{align} & \int_{F^2}^{\rm st} \int_{F^2}^{\rm st} \int_{Q_0 \backslash G^1} \int_{G^1} \int_{M_{X}} \int_{X_+^2} \langle \pi(g)f, f^\prime \rangle (\omega_{\psi}(hg, t) \phi)(A_0+b) \nonumber\\ &\quad \times \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+b)} \Lambda(t)\psi \bigg(\mathrm{tr} \begin{pmatrix} s_2 & t_2\\ s_1 & t_1 \end{pmatrix} \bigg(S_0 \begin{pmatrix} \langle x_{-2}, b_1 \rangle & \langle x_{-2}, b_2 \rangle\\ \langle x_{-1}, b_1 \rangle & \langle x_{-1}, b_2 \rangle \end{pmatrix} -X\bigg) \bigg) \nonumber\\ &\quad \times db \, dh \, {dx} \, dg \, dt \, ds_2 \,dt_2 \,ds_1 \, dt_1 . \end{align}

Now we claim that we may define the stable integral

\begin{align*} & \int_{F^2}^{\rm st} \int_{F^2}^{\rm st} \int_{X_+^2} \langle \pi(g)f, f^\prime \rangle (\omega_{\psi}(hg, t) \phi)(A_0+b) \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+b)} \\ &\quad \times\Lambda(t) \psi \bigg(\mathrm{tr} \begin{pmatrix} s_2 & t_2\\ s_1 & t_1 \end{pmatrix}\bigg(S_0 \begin{pmatrix} \langle x_{-2}, b_1 \rangle & \langle x_{-2}, b_2 \rangle\\ \langle x_{-1}, b_1 \rangle & \langle x_{-1}, b_2 \rangle \end{pmatrix} -X\bigg)\bigg) db \, ds_2 \,dt_2 \,ds_1 \, dt_1 \end{align*}

and we may choose a sufficiently large compact open subgroup $F_i$ of $F$ ($1 \leq i \leq 4$) so that it depends only on $\psi$ and $\int _{F^2}^{\rm st} \int _{F^2}^{\rm st}\cdots = \int _{F_1} \int _{F_2}\int _{F_3}\int _{F_4}\cdots$. This claim easily follows from the following lemma in the one-dimensional case.

Lemma 7.1 Let $f$ be a locally constant function on $F$ which is in $L^1(F)$. Then there exists a compact open subgroup $F_0$ of $F$ such that for any compact open subgroups $F^\prime$ and $F^{\prime \prime }$ of $F$ containing $F_0$, we have

(7.1.4)\begin{equation} \int_{F^\prime} \int_{F} f(x) \psi(xy) \, {dx} \, {dy} = \int_{F^{\prime \prime}} \int_{F} f(x) \psi(xy) \, {dx} \, {dy}. \end{equation}

Proof. Suppose that $\psi$ is trivial on $F_0 : =\varpi ^{m}\mathcal {O}_F$ and not trivial on $\varpi ^{m-1}\mathcal {O}_F$. Put $F^\prime = \varpi ^{m^\prime } \mathcal {O}_F$ with $m^{\prime } \leq m$. Then we may write the left-hand side of (7.1.4) as

(7.1.5)\begin{equation} \int_{F^\prime} \int_{F \setminus \mathcal{O}} f(x) \psi(xy)\, {dx} \, {dy} +\int_{F^\prime}\int_{\mathcal{O}} f(x) \psi(xy) \, {dx} \, {dy}. \end{equation}

The first integral of (7.1.5) converges absolutely. Hence by interchanging the order of integration, it is equal to

\[ \int_{F \setminus \mathcal{O}} \int_{F^\prime} f(x) \psi(xy)\, {dy} \, {dx} =\int_{F \setminus \mathcal{O}}f(x)\bigg(\int_{F^\prime}\psi(xy)\,{dy}\bigg){dx}=0 \]

since $y \mapsto \psi (xy)$ is a non-trivial character of $F^\prime$ for each $x \in F\setminus \mathcal {O}$. As for the second integral of (7.1.5), we have

\begin{align*} & \int_{F^\prime}\int_{\mathcal{O}} f(x) \psi(xy)\,{dx}\,{dy} =\int_{\varpi^{m} \mathcal{O}} \int_{\mathcal{O}} f(x) \psi(xy)\,{dx}\,{dy} +\int_{\varpi^{m^\prime} \mathcal{O} \setminus \varpi^{m} \mathcal{O}}f(x) \bigg(\int_{\mathcal{O}}\psi(xy)\,{dy}\bigg){dx}, \end{align*}

where the inner integral of the second integral vanishes as above. Thus, the left-hand side of (7.1.4) is equal to

\[ \int_{\varpi^{m} \mathcal{O}} \int_{\mathcal{O}} f(x) \psi(xy)\,{dx}\,{dy}. \]

Similarly the right-hand side of (7.1.4) becomes as above, and our claim follows.

By Lemma 7.1, we see that (7.1.3) is equal to

\begin{align*} & \int_{N \backslash G^1} \int_{G^1} \int_{M_{X}} \int_{F^2}^{\rm st} \int_{F^2}^{\rm st} \int_{X_+^2} \langle \pi(g)f, f^\prime \rangle (\omega_{\psi}(hg, t) \phi)(A_0+b) \\ &\quad \times \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+b)} \Lambda(t) \psi \bigg(\mathrm{tr} \begin{pmatrix} s_2 & t_2\\ s_1 & t_1 \end{pmatrix}\bigg(S_0 \begin{pmatrix} \langle x_{-2}, b_1 \rangle & \langle x_{-2}, b_2 \rangle\\ \langle x_{-1}, b_1 \rangle & \langle x_{-1}, b_2 \rangle \end{pmatrix} -X\bigg)\bigg)\\ &\quad \times db \, dh \, {dx} \, dg \, dt \, ds_2 \,dt_2 \,ds_1 \, dt_1 . \end{align*}

Then applying the Fourier inversion, we get

(7.1.6) \begin{align} & \int_{N \backslash G^1} \int_{G^1} \int_{M_{X}} \langle \pi(g)f, f^\prime \rangle (\omega_{\psi}(hg, t) \phi)(A_0+B_0) \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+B_0)} \Lambda(t) \, db \, dh \, {dx} \, dg \, dt, \end{align}

where $B_0 = (0, 0,( {x_{21}}/{2})x_1+({x_{11}}/{2}) x_2, -({x_{22}}/{2d})x_1-( {x_{12}}/{2d}) x_2)$ and $x_0=A_0+B_0$. By [Liu16, Proposition 3.1], for a sufficiently large compact open subgroup $N_0$ of $N$, we have

\[ \int_{M_X} \int^{\rm st}_{N} f(nt) \chi(nt)\,dn \,dt = \int_{N_0} \int_{M_X} f(nt) \chi(nt)\,dn \,dt \]

and, thus, we may define

\[ \int_{N}^{\rm st} \int_{M_X} f(nt) \chi(nt) \,dn \,dt. \]

Further, we note a simple fact that we have

\[ \int_{G} g(h) \,dh = \int_{N \backslash G} \int_{N}^{\rm st} g(nh) \, dn \,dh \]

when both sides are defined. Thus, (7.1.6) is equal to

\begin{align*} & \int_{N \backslash G^1} \int_{N \backslash G^1} \int_{M_{X}} \int_{N}^{\rm st} \langle \pi(g)f, f^\prime \rangle \\ &\quad \times (\omega_{\psi}(hg, t) \phi)(A_0+B_0) \overline{(\omega_{\psi}(h,1)\phi^\prime)(A_0+B_0)} \Lambda(t) \, db \, dh \, {dx} \, dg \, dt . \end{align*}

Then the same computation as that to get (3.1.17) from (3.1.16) may be applied to the above integral, and, thus, we see that our integral is equal to

\begin{align*} & \int_{N\backslash G^1}\int_{N\backslash G^1} \alpha_{\Lambda, \psi_{S}}(\pi_v(h)f,\pi_v(h^\prime)f^\prime) (\omega_{\psi}(h,1)\phi)(x_0) \overline{(\omega_{\psi}(h^\prime,1)\phi^\prime)(x_0)}\,dh\,dh^\prime. \end{align*}

Hence, the identity (7.1.2) holds when $B_{\xi, \Lambda,\psi } \not \equiv 0$.

7.2 Proof of Theorem 1.2 when $B_{\xi, \Lambda,\psi } \equiv 0$

First we note the following proposition concerning the non-vanishing of the $L$-values.

Proposition 7.3 Let $\pi$ be an irreducible cuspidal tempered automorphic representation of $G_D(\mathbb {A})$ with trivial central character. If $G_D \simeq G$ and $\pi$ is a theta lift from $\mathrm {GSO}_{3,1}$, then $L(s, \pi, \mathrm {std} \otimes \chi _E)$ has a simple pole at $s=1$. Otherwise $L(s, \pi, \mathrm {std} \otimes \chi _E)$ is holomorphic and non-zero at $s=1$.

Proof. Suppose that $G_D \simeq G$, i.e. $D$ is split. Then there exists an irreducible cuspidal globally generic automorphic representation $\pi _0$ of $G(\mathbb {A})$ such that $\pi$ and $\pi _0$ are nearly equivalent. Then our claim follows from [Yam14, Lemma 10.2] and [Sha81, Theorem 5.1].

Suppose that $D$ is not split. Let us take a quadratic extension $E_0$ of $F$ such that $\pi$ has $(E_0, \Lambda _0)$-Bessel period for some character $\Lambda _0$ of $\mathbb {A}_{E_0}^\times / E_0^\times$. Then by Theorem 1.1(i), we see that there exists an irreducible cuspidal tempered automorphic representation $\pi _0$ of $G(\mathbb {A})$ such that for a sufficiently large finite set $S$ of places of $F$ containing all archimedean places, $\pi _v, \pi _{0,v}$ are unramified and $\mathrm {BC}_{E_0 / F}(\pi _v) \simeq \mathrm {BC}_{E_0 / F}(\pi _{0,v})$ for $v\not \in S$. This implies that

\begin{align*} & L^S(s, \pi_0, \mathrm{std} \otimes \chi_{E_0} \chi_E) L^S(s, \pi_0, \mathrm{std} \otimes \chi_E) =L^S(s, \pi, \mathrm{std} \otimes \chi_{E_0} \chi_E) L^S(s, \pi, \mathrm{std} \otimes \chi_E). \end{align*}

From the case when $G_D \simeq G$, the left-hand side of this identity is not zero at $s=1$, and, thus, so is the right-hand side, which possibly has a pole at $s=1$.

Suppose that $L^S(s, \pi, \mathrm {std} \otimes \chi _{E_0 / F} \chi _E)$ has a pole at $s=1$. We may take a quadratic extension $E_1 \subset D$ of $F$ such that $\chi _{E_1} = \chi _{E_0} \chi _E$. Then by Yamana [Yam14, Lemma 10.2], $\pi$ is a theta lift from $\mathrm {GSU}_{1,D}$, which is a similitude quaternion unitary group of degree one defined by an element in $E_1$ as in (2.1.12). In this case, $\pi$ is not tempered and, thus, it contradicts our assumption on $\pi$. Thus, $L^S(s, \pi, \mathrm {std} \otimes \chi _{E_0 / F} \chi _E)$ is holomorphic at $s=1$. Further, by an argument similar to that for $L^S(s, \pi, \mathrm {std} \otimes \chi _{E_0 / F} \chi _E)$, we see that $L^S(s, \pi, \mathrm {std} \otimes \chi _E)$ is holomorphic. Therefore, it is holomorphic and non-zero at $s=1$.

Suppose that $B_{\xi, \Lambda,\psi } \equiv 0$ on $V_\pi$. We shall show that the right-hand side of (1.6.2) is zero. If $L(\frac {1}{2}, \pi \times \mathcal {AI} (\Lambda )) = 0$, then there is nothing to prove. Hence, we may suppose that $L(\frac {1}{2}, \pi \times \mathcal {AI} (\Lambda )) \ne 0$. Then we shall show that for some place $v$ of $F$, we have $\alpha _{\Lambda _v, \psi _{\xi, v}}\equiv 0$ on $\pi _v$.

Assume the contrary, i.e. $\alpha _{\Lambda _v, \psi _{\xi, v}} \not \equiv 0$ on $\pi _v$ for any $v$. Let us denote by $\pi _+^{B, {\rm loc}}$ the unique irreducible constituent of $\pi |_{G_D(\mathbb {A})^+}$ such that $\alpha _{\Lambda _v, \psi _{\xi, v}} \not \equiv 0$ on $\pi _{+,v}^{B, \mathrm {loc}}$ for any $v$. From our assumption $\alpha _{\Lambda _v, \psi _{\xi, v}} \not \equiv 0$ on $\pi _v$ and Corollary 7.2, we see that $\alpha _{\Lambda _v^{-1}, \psi _{X, v}} \not \equiv 0$ on the theta lift $\theta _{\psi _v, D}(\pi _v)$ of $\pi _v$ to $\mathrm {GSU}_{3,D}(F_v)$ and $Z_v(\phi _v, f_v, \pi ) \ne 0$ for some $f_v \in \pi _v$ and $\phi _v \in \mathcal {S}(Z_{D, +}(F_v))$. Since $\pi ^\prime$ is nearly equivalent to $\pi$, we have $L(1, \pi, \mathrm {std} \otimes \chi _E) \ne 0$. Therefore, the theta lift $\theta _{\psi, D}(\pi _+^{B, \mathrm {loc}})$ of $\pi _+^{B, \mathrm {loc}}$ to $\mathrm {GSU}_{3,D}(\mathbb {A})$ is non-zero by Yamana [Yam14, Theorem 10.3], which states that the non-vanishing of local theta lifts at all places together with the non-vanishing of the $L$-value implies the non-vanishing of the global theta lift. We note that actually in [Yam14, Theorem 10.3], there is an assumption that $D$ is not split at real places, which was necessary to ensure that the non-vanishing of the local theta lift implies $Z_v(\phi _v, f_v, \pi ) \ne 0$ for some $f_v \in \pi _v$ and $\phi _v \in \mathcal {S}(Z_{D, +}(F_v))$. Since the non-vanishing of $Z_v(\phi _v, f_v, \pi )$ for some $f_v$ and $\phi _v$ is shown in our case by the argument above, we may apply [Yam14, Theorem 10.3] regardless of the assumption.

Recall that from the proof of Theorem 1.1(i), $\theta _{\psi, D}(\pi _+^{B, \mathrm {loc}})$ is tempered. Let us regard $\theta _{\psi, D}(\pi _+^{B, \mathrm {loc}})$ as automorphic representations of $\mathrm {GU}_{4, \varepsilon }$. By the uniqueness of the Bessel model for $\mathrm {GU}_{4, \varepsilon }$ proved in [FM22, Proposition A.1], there uniquely exists an irreducible constituent $\tau$ of $\theta _{\psi, D}(\pi _+^{B, \mathrm {loc}})|_{\mathrm {U}(4)}$ such that $\tau$ has the local $(X, \Lambda _v^{-1},\psi _v)$-Bessel model at any place $v$.

On the other hand, we note $L(1 / 2, \tau \times \Lambda ^{-1}) \ne 0$ since $L(\frac {1}{2}, \pi \times \mathcal {AI} (\Lambda )) \ne 0$. Then by [FM22, Theorem 1.2], there exists an irreducible cuspidal automorphic representation $\tau ^\prime$ of $\mathrm {U}(V_0)$ with four-dimensional hermitian space $V_0$ over $E$ such that $\tau ^\prime$ has $(X, \Lambda _v,\psi _v)$-Bessel period. Then we know that $\tau$ and $\tau ^\prime$ have the same $L$-parameter, in particular, $\tau _v \simeq \tau _v^\prime$ when $v$ is split. At a non-split place $v$, by the uniqueness of an element of the tempered $L$-packet which has the same Bessel period due to Beuzart-Plessis [Beu16, Beu20], we see that $\mathrm {U}(V_0) \simeq \mathrm {U}(J_D)$ and $\tau \simeq \tau ^\prime$. Moreover, by Mok [Mok15], we have $\tau = \tau ^\prime$. Therefore, $\tau = \tau ^\prime$ has $(X, \Lambda ^{-1},\psi )$-Bessel period, and this implies that $\theta _{\psi, D}(\pi _+^{B, \mathrm {loc}})$ also has $(X, \Lambda ^{-1},\psi )$-Bessel period. Then Propositions 3.1 and 3.2 show that $\pi$ has $(E,\Lambda )$-Bessel period, and this is a contradiction. Thus, (1.6.2) holds when $B_{\xi, \Lambda,\psi } \equiv 0$ on $V_\pi$.

8. Generalized Böcherer conjecture

In this section we prove the generalized Böcherer conjecture. In fact, we shall prove Theorem 8.1 below, which is more general than Theorem 1.4 stated in the introduction.

8.1 Temperedness condition

In order to apply Theorem 1.2 to holomorphic Siegel cusp forms of degree two, we need to verify the temperedness for corresponding automorphic representations.

Proposition 8.1 Suppose that $F$ is totally real. Let $\tau$ be an irreducible cuspidal automorphic representation of $G_D(\mathbb {A})$ with a trivial central character such that $\tau _v$ is a discrete series representation for every real place $v$ of $F$. Suppose moreover that $\tau$ is not CAP (cuspidal representation associated to parabolic subgroup). Then $\tau$ is tempered.

Remark 8.1 When $D$ is split, i.e. $G_D \simeq G$, Weissauer [Wei09] proved that $\tau _v$ is tempered at a place $v$ when $\tau _v$ is unramified. Moreover, when $\tau _v$ is a holomorphic discrete series representation at each archimedean place $v$, Jorza [Jor13] showed the temperedness at finite places not dividing $2$.

Proof. First suppose that $G_D \simeq G$. Let $\Pi$ denote the functorial lift of $\tau$ to $\mathrm {GL}_4(\mathbb {A})$ established by Arthur [Art13] (see also Cai, Friedberg and Kaplan [CFK18]).

When $\Pi$ is not cuspidal, since $\tau$ is not CAP, $\Pi$ is of the form $\Pi =\Pi _1 \boxplus \Pi _2$ with irreducible cuspidal automorphic representations $\Pi _i$ of $\mathrm {GL}_2(\mathbb {A})$. Since $\tau _v$ is a discrete series representation for any real place $v$, $\Pi _{i,v}$ is also a discrete series representation. Then $\Pi _i$ is tempered by [Bla06] and, thus, the Langlands parameter of $\Pi _v$ is tempered at all places $v$ of $F$. Hence, $\tau$ is tempered.

Suppose that $\Pi$ is cuspidal. Then by Raghuram and Sarnobat [RS18, Theorem 5.6], $\Pi _v$ is tempered and cohomological at any real place $v$. Let us take an imaginary quadratic extension $E$ of $F$ such that the base change lift $\mathrm {BC}(\Pi )$ of $\Pi$ to $\mathrm {GL}_4(\mathbb {A}_E)$ is cuspidal. Note that $\mathrm {BC}(\Pi )$ is cohomological and that $\mathrm {BC}(\Pi )^\vee \simeq \mathrm {BC}(\Pi ^\vee )\simeq \mathrm {BC}(\Pi ) \simeq \mathrm {BC}(\Pi )^\sigma$. Then Caraiani [Car12, Theorem 1.2] shows that $\mathrm {BC}(\Pi )$ is tempered at all finite places. This implies that $\Pi _v$ is also tempered for any finite place $v$. Thus, $\tau$ is tempered.

Now let us consider the case when $D$ is not split. Since $\tau$ is not CAP, by Proposition 4.1, there exists an irreducible cuspidal automorphic representation $\tau ^\prime$ of $G(\mathbb {A})$ and a quadratic extension $E_0$ of $F$ such that $\tau ^\prime$ is $G^{+, E_0}$-locally equivalent to $\tau$. Moreover $\tau$ is tempered if and only if $\tau ^\prime$ is tempered. By [LPTZ03, Mœg89, Pau98, Pau00], $\tau ^\prime _v$ is a discrete series representation at any real place $v$. Then the temperedness of $\tau ^\prime$ follows from the split case. Hence, $\tau$ is also tempered.

As an application of Proposition 8.1, the following corollary holds.

Corollary 8.1 Suppose that $F$ is totally real. Let $\tau$ be an irreducible cuspidal globally generic automorphic representation of $G(\mathbb {A})$ such that $\tau _v$ is a discrete series representation at any real place $v$. Then $\tau$ is tempered and, hence, the explicit formula (6.2.3) for the Whittaker periods holds for any non-zero decomposable vector in $V_\tau$.

Proof. Recall that the functorial lift $\Pi$ of $\tau$ to $\mathrm {GL}_4(\mathbb {A})$ is cuspidal or an isobaric sum of irreducible cuspidal automorphic representations of $\mathrm {GL}_2$ by [CKPS04]. In particular, $\tau$ is not CAP by Arthur [Art13]. Then by Proposition 8.1, $\tau$ is tempered and our claim follows from Theorem 6.3.

8.2 Vector-valued Siegel cusp forms and Bessel periods

Let $\mathfrak H_2$ be the Siegel upper half-space of degree two, i.e. the set of two-by-two symmetric complex matrices whose imaginary parts are positive definite. Then the group $G(\mathbb {R})^+=\{g\in G(\mathbb {R}):\nu (g)>0\}$ acts on $\mathfrak H_2$ by

\[ g\langle Z\rangle=(AZ+B)(CZ+D)^{-1}\quad \text{for $g=\begin{pmatrix} A & B\\ C & D \end{pmatrix}\in G(\mathbb{R})^+$ and $Z\in\mathfrak H_2$} \]

and the factor of automorphy $J(g,Z)$ is defined by

\[ J(g,Z)=CZ+D. \]

For an integer $N\ge 1$, let

\[ \Gamma_0(N)=\bigg\{\gamma\in G^1(\mathbb{Z}):\gamma=\begin{pmatrix} A & B\\ C & D\end{pmatrix},\ C\equiv 0\ {\rm (mod}\ N\mathbb{Z})\bigg\}. \]

8.2.1 Vector-valued Siegel cusp forms

Let $(\varrho, V_\varrho )$ be an algebraic representation of $\mathrm {GL}_2(\mathbb {C})$. Then a holomorphic mapping $\Phi :\mathfrak H_2\to V_\varrho$ is a Siegel cusp form of weight $\varrho$ with respect to $\Gamma _0(N)$ when $\Phi$ vanishes at the cusps and satisfies

(8.2.1)\begin{equation} \Phi(\gamma\langle Z\rangle)=\varrho(J(\gamma, Z))\Phi(Z)\quad \text{for $\gamma\in \Gamma_0(N)$ and $Z\in\mathfrak H_2$}. \end{equation}

We denote by $S_\varrho (\Gamma _0(N))$ the complex vector space of Siegel cusp forms of weight $\varrho$ with respect to $\Gamma _0(N)$. Then $\Phi \in S_\varrho (\Gamma _0(N))$ has a Fourier expansion

\[ \Phi(Z)=\sum_{T>0} a(T,\Phi)\exp[2\pi\sqrt{-1}\,\mathrm{tr}(TZ)] \quad\text{where $Z\in\mathfrak H_2$ and $a(T,\Phi)\in V_\varrho$}. \]

Here $T$ runs over positive-definite two-by-two symmetric matrices which are semi-integral, i.e. $T$ is of the form $T=\big (\begin{smallmatrix}a & b/ 2\\ b/ 2 & c\end{smallmatrix}\big )$, $a,b,c\in \mathbb {Z}$. We note that (8.2.1) implies

(8.2.2)\begin{equation} a(\varepsilon \,T\,{}^t\varepsilon,\Phi)=\varrho(\varepsilon) a(T,\Phi) \quad\text{for $\varepsilon\in\mathrm{GL}_2(\mathbb{Z})$}. \end{equation}

From now until the end of this paper, we assume $\varrho$ to be irreducible. It is well known that the irreducible algebraic representations of $\mathrm {GL}_2(\mathbb {C})$ are parametrized by

(8.2.3)\begin{equation} \mathbb{L}=\{(n_1,n_2)\in\mathbb{Z}^2: n_1\ge n_2\}. \end{equation}

Namely the parametrization is given by assigning

\[ \varrho_\kappa:=\mathrm{Sym}^{n_1-n_2}\otimes {\det}^{n_2} \quad\text{to $\kappa=(n_1,n_2)\in\mathbb{L}$}. \]

Suppose that $\varrho =\varrho _\kappa$ with $\kappa =(n+k,k)\in \mathbb {L}$. Then we realize $\varrho$ concretely by taking its space of representation $V_{\varrho }$ to be $\mathbb {C}[X,Y]_{n}$, the space of degree $n$ homogeneous polynomials of $X$ and $Y$, where the action of $\mathrm {GL}_2(\mathbb {C})$ is given by

\[ \varrho(g)P(X,Y)=(\det g)^k\cdot P((X,Y)g)\quad \text{for $g\in\mathrm{GL}_2(\mathbb{C})$ and $P\in\mathbb{C}[X,Y]_n$}. \]

Let us define a bilinear form

\[ \mathbb{C}[X,Y]_n\times \mathbb{C}[X,Y]_n\ni(P,Q)\mapsto(P,Q)_n\in \mathbb{C} \]

by

(8.2.4)\begin{equation} (X^iY^{n-i}, X^j Y^{n-j})_n= \begin{cases} \displaystyle{(-1)^i \begin{pmatrix}n\\ i\end{pmatrix}} & \text{if $i+j=n$}; \\ 0 & \text{otherwise.} \end{cases} \end{equation}

Then we have

(8.2.5)\begin{equation} (\varrho(g)P,\varrho(g)Q)_n=(\det g)^{n+2k}(P,Q)_n \quad\text{for $g\in\mathrm{GL}_2(\mathbb{C})$.} \end{equation}

We define a positive-definite hermitian inner product $\langle \,{,}\,\rangle _\varrho$ on $V_\varrho$ by

(8.2.6)\begin{equation} \langle P,Q\rangle_\varrho:=(P,\varrho(w_0)\overline{Q})_n \quad\text{where $w_0=\begin{pmatrix}0 & 1\\-1 & 0\end{pmatrix}$.} \end{equation}

Here $\overline {Q}$ denotes the polynomial obtained from $Q$ by taking the complex conjugates of its coefficients. Then (8.2.5) implies that we have

(8.2.7)\begin{equation} \langle\varrho(g)v,w\rangle_{\varrho}=\langle v,\varrho({}^t\bar{g})w\rangle_{\varrho}\quad \text{for $g\in\mathrm{GL}_2(\mathbb{C})$ and $v,w\in V_\varrho$}. \end{equation}

In particular, the hermitian inner product $\langle \,{,}\,\rangle _\varrho$ is $\mathrm {U}_2(\mathbb {R})$-invariant. Then for $\Phi,\Phi ^\prime \in S_{\varrho }(\Gamma _0(N))$, we define the Petersson inner product $\langle \Phi,\Phi ^\prime \rangle _{\varrho }$ by

(8.2.8)\begin{equation} \langle\Phi,\Phi^\prime\rangle_{\varrho}= \frac{1}{[\mathrm{Sp}_2(\mathbb{Z}): \Gamma_0(N)]}\int_{\Gamma_0(N)\backslash \mathfrak H_2} \langle\Phi(Z),\Phi^\prime(Z)\rangle_{\varrho}(\det Y)^{k-3}\, dX\, dY, \end{equation}

where $X=\mathrm {Re}(Z)$ and $Y=\mathrm {Im}(Z)$. The space $S_{\varrho }(\Gamma _0(N))$ has a natural orthogonal decomposition with respect to the Petersson inner product

\[ S_{\varrho}(\Gamma_0(N))=S_{\varrho}(\Gamma_0(N))^{\mathrm{old}}\oplus S_{\varrho}(\Gamma_0(N))^{\mathrm{new}} \]

into the oldspace and the newspace in the sense of Schmidt [Sch05, 3.3]. We note that when $n$ is odd, we have $S_\varrho (\Gamma _0(N))=\{0\}$ for $\varrho$ with $\kappa =(n+k,k)$ by (8.2.1) since $-1_4\in \Gamma _0(N)$.

8.2.2 Adelization

Given $\Phi \in S_\varrho (\Gamma _0(N))$, its adelization $\varphi _\Phi :G(\mathbb {A})\to V_\varrho$ is defined as follows (cf. [Sah15, 3.1] and [Sch05, 3.2]). For each prime number $p$, let us define a compact open subgroup $P_{1,p}(N)$ of $G(\mathbb {Q}_p)$ by

\[ P_{1,p}(N):=\bigg\{ g\in G(\mathbb{Z}_p): g=\begin{pmatrix}A & B\\C & D\end{pmatrix}, C\equiv 0\ {\rm (mod}\ N\,\mathbb{Z}_p)\bigg\}. \]

Then we define a mapping $\varphi _{\Phi }:G(\mathbb {A}) \to V_{\varrho }$ by

(8.2.9)\begin{equation} \varphi_{\Phi}(g)=\nu(g_\infty)^{k+r}\varrho(J(g_\infty, \sqrt{-1}\, 1_2))^{-1} \Phi(g_\infty\langle\sqrt{-1}\, 1_2\rangle) \end{equation}

when

\[ g=\gamma g_\infty k_0\quad \text{with $\gamma\in G(\mathbb{Q})$, $g_\infty\in G(\mathbb{R})^+$ and $k_0\in\prod_{p<\infty}P_{1,p}(N)$}. \]

Let $L$ be any non-zero linear form on $V_\varrho$. Then $L(\varphi _\Phi ):G(\mathbb {A})\to \mathbb {C}$ defined by $L(\varphi _\Phi )(g)=L(\varphi _\Phi (g))$ is a scalar-valued automorphic form on $G(\mathbb {A})$. Let $V(\Phi )$ denote the space generated by right $G(\mathbb {A})$-translates of $L(\varphi _\Phi )$. Then $V(\Phi )$ does not depend on the choice of $L$ and we denote by $\pi (\Phi )$ the right regular representation of $G(\mathbb {A})$ on $V(\Phi )$. Note that the central character of $\pi (\Phi )$ is trivial.

We recall that for scalar-valued automorphic forms $\phi$, $\phi ^\prime$ on $G(\mathbb {A})$ with a trivial central character, their Petersson inner product $\langle \phi,\phi ^\prime \rangle$ is defined by

\[ \langle \phi,\phi^\prime\rangle =\int_{Z_G(\mathbb{A}) G(\mathbb{Q}) \backslash G(\mathbb{A})}\phi(g)\overline{\phi^\prime(g)}\,dg, \]

where $Z_G$ denotes the center of $G$ and $dg$ is the Tamagawa measure.

Lemma 8.1 Let $L$ be a non-zero linear form on $V_\varrho$. Take $v^\prime \in V_\varrho$ such that $L(v)=\langle v,v^\prime \rangle _\varrho$ for any $v\in V_\varrho$.

Then we have

\[ \langle L(\varphi_\Phi),L(\varphi_\Phi)\rangle=C(v^\prime)\cdot \langle\Phi,\Phi\rangle_\varrho\quad\text{for any $\Phi\in S_\varrho(\Gamma_0(N))$,} \]

where

(8.2.10)\begin{equation} C(v^\prime)=\frac{\mathrm{Vol}(Z_G(\mathbb{A}) G(\mathbb{Q}) \backslash G(\mathbb{A}))}{ \mathrm{Vol}(\mathrm{Sp}_2(\mathbb{Z})\backslash \mathfrak H_2)}\cdot\frac{\langle v^\prime,v^\prime\rangle_\varrho}{ \dim V_\varrho}. \end{equation}

Proof. Let $K_\infty =\mathrm {U}_2(\mathbb {R})$. We identify $K_\infty$ as a subgroup of $\mathrm {Sp}_2(\mathbb {R})$ via

\[ K_\infty\ni A+\sqrt{-1}\, B\mapsto \begin{pmatrix} A & -B\\ B & A\end{pmatrix}\in \mathrm{Sp}_2(\mathbb{R}). \]

Let $dk$ be the Haar measure on $K_\infty$ such that $\mathrm {Vol}(K_\infty, dk)=1$. Then by the Schur orthogonality relations, we have

\[ \int_{K_\infty}L(\varrho(k)^{-1}v)\cdot\overline{L(\varrho(k)^{-1}w)}\,dk =\frac{\langle v,w\rangle_\varrho\cdot\langle v^\prime,v^\prime\rangle_\varrho}{\dim V_\varrho}. \]

On the other hand, it is easily seen that for $\Phi \in S_\varrho (\Gamma _0(N))$, we have

\[ \frac{\langle\Phi,\Phi\rangle_\varrho}{\mathrm{Vol} (\mathrm{Sp}_2(\mathbb{Z})\backslash \mathfrak H_2)} =\frac{\langle\varphi_\Phi,\varphi_\Phi\rangle_\varrho}{\mathrm{Vol} (Z_G(\mathbb{A}) G(\mathbb{Q}) \backslash G(\mathbb{A}))}, \]

where

\[ \langle\varphi_\Phi,\varphi_\Phi\rangle_\varrho:= \int_{Z_G(\mathbb{A}) G(\mathbb{Q}) \backslash G(\mathbb{A})} \langle\varphi_\Phi(g),\varphi_\Phi(g)\rangle_\varrho \,dg. \]

Hence,

\begin{align*} \langle\Phi,\Phi\rangle_\varrho &= C(v^\prime)^{-1} \int_{Z_G(\mathbb{A}) G(\mathbb{Q}) \backslash G(\mathbb{A})} \int_{K_\infty}| L(\varrho(k)^{-1}\varphi_\Phi(g))|^2\, dk\,dg \\ &= C(v^\prime)^{-1}\int_{K_\infty} \int_{Z_G(\mathbb{A}) G(\mathbb{Q}) \backslash G(\mathbb{A})} | L(\varphi_\Phi(gk))|^2\,dg\,dk \\ &= C(v^\prime)^{-1}\cdot\langle L(\varphi_\Phi),L(\varphi_\Phi)\rangle_\varrho. \end{align*}

8.2.3 Bessel periods of vector-valued Siegel cusp forms

Let $E$ be an imaginary quadratic field of $\mathbb {Q}$ and $-D_E$ its discriminant. We put

(8.2.11) \begin{equation} S_E:=\begin{cases} \,\begin{pmatrix} 1 & 0\\ 0 & D_E/ 4 \end{pmatrix} & \text{when $D_E\equiv 0\pmod{4}$};\\ \begin{pmatrix} 1 & 1/ 2\\ 1/ 2 & (1+D_E)/ 4 \end{pmatrix} & \text{when $D_E\equiv -1\pmod{4}$}. \end{cases} \end{equation}

Given $S=S_E$ as above, we define $T_S$, $N$ and $\psi _{S}$ as in § 2.3.1. Then $T_S(\mathbb {Q})\simeq E^\times$.

Let $\Lambda$ be a character of $T_S(\mathbb {A})$ which is trivial on $\mathbb {A}^\times T_S(\mathbb {Q})$. Let $\psi$ be the unique character of $\mathbb {A}/ \mathbb {Q}$ such that $\psi _\infty (x)=e^{-2\pi \sqrt {-1}\, x}$ and the conductor of $\psi _\ell$ is $\mathbb {Z}_\ell$ for any prime number $\ell$.

Then for a scalar-valued automorphic form $\phi$ on $G(\mathbb {A})$ with a trivial central character, we define its $(S, \Lambda, \psi )$-Bessel period $B_{S,\Lambda, \psi }(\phi )$ by (2.3.1) with the Haar measures $du$ on $N(\mathbb {A})$ and $dt=dt_\infty \, dt_f$ on $T_S(\mathbb {A})=T_S(\mathbb {R}) \times T_S(\mathbb {A}_f)$ are taken so that $\mathrm {Vol}(N(\mathbb {Q})\backslash N(\mathbb {A}), du)=1$ and

\[ \mathrm{Vol}(\mathbb{R}^\times\backslash T_S(\mathbb{R}), dt_\infty)= \mathrm{Vol}(T_S(\mathbb{Z}_p), dt_f)=1. \]

Then we note that

\[ \mathrm{Vol}(\mathbb{A}^\times T_S(\mathbb{Q}) \backslash T_S(\mathbb{A}),dt) = \frac{2h_E}{w(E)} = D_E^{1 / 2} \cdot L(1, \chi_E). \]

For a $V_\varrho$-valued automorphic form $\varphi$ with a trivial central character, it is clear that for a linear form $L:V_\varrho \to \mathbb {C}$ we have

(8.2.12)\begin{equation} B_{S,\Lambda, \psi}(L(\varphi))=L\bigg[\int_{\mathbb{A}^\times T_S(\mathbb{Q}) \backslash T_S(\mathbb{A})}\int_{N(\mathbb{Q})\backslash N(\mathbb{A})} \Lambda (t)^{-1}\psi_S (u)^{-1} \varphi(tu)\, dt\, du\bigg]. \end{equation}

Recall that we may identify the ideal class group $\mathrm {Cl}_E$ of $E$ with the quotient group

\[ T_S(\mathbb{A})/ T_S(\mathbb{Q})T_S(\mathbb{R}) T_S(\hat{\mathbb{Z}}). \]

Let $\{t_c: c\in \mathrm {Cl}_E\}$ be a set of representatives of $\mathrm {Cl}_E$ such that $t_c\in \prod _{p<\infty }T(\mathbb {Q}_p)$. We let $t_c=\gamma _c\,m_c\,\kappa _c$ with $\gamma _c\in \mathrm {GL}_2(\mathbb {Q})$, $m_c\in \{g\in \mathrm {GL}_2(\mathbb {R}):\det g>0\}$, $\kappa _c\in \prod _{p<\infty }\mathrm {GL}_2(\mathbb {Z}_p)$. Let $S_c=(\det \gamma _c)^{-1}\cdot {}^t\gamma _c S\gamma _c$. Then the set $\{S_c: c\in \mathrm {Cl}_E\}$ is a set of representatives for the $\mathrm {SL}_2(\mathbb {Z})$-equivalence classes of primitive semi-integral positive-definite two-by-two symmetric matrices of discriminant $D_E$.

Thus, when $\varphi =\varphi _\Phi$ for $\Phi \in S_\varrho (\Gamma _0(N))$ and $\Lambda$ is a character of $\mathrm {Cl}_E$, we may write (8.2.12) as

(8.2.13)\begin{equation} B_{S,\Lambda, \psi}(L(\varphi_\Phi))=2\cdot e^{-2\pi\mathrm{tr}(S)}\cdot L(B_\Lambda(\Phi;E)), \end{equation}

where

(8.2.14)\begin{equation} B_\Lambda (\Phi; E):= w(E)^{-1}\cdot \pi_\varrho\bigg(\sum_{c\in\mathrm{Cl}_E} \Lambda(c)^{-1} \cdot a(S_c,\Phi)\bigg) \end{equation}

is the vector-valued $(S, \Lambda, \psi )$-Bessel period where

(8.2.15)\begin{equation} \pi_\varrho=\int_{T_S^1(\mathbb{R})}\varrho(t)\,dt \quad \text{with $T_S^1=\mathrm{SL}_2\cap T_S$, $\mathrm{Vol}(T_S^1(\mathbb{R}),dt)=1$} \end{equation}

(e.g. Dickson et al. [DPSS20, Proposition 3.5] and Sugano [Sug85, (1-26)]).

Remark 8.2 (An erratum to [FM17])

The definition of $B(\Phi ;E)$ in the vector-valued case in [FM17, Theorem 5] should be replaced by (8.2.14). The statement and the proof of [FM17, Theorem 5] remain valid.

Suppose that $\varrho =\varrho _\kappa$ where $\kappa =(2r+k,k) \in \mathbb {L}$. We define $Q_{S,\varrho }\in \mathbb {C}[X,Y]_{2r}$ by

(8.2.16)\begin{equation} Q_{S,\varrho}(X,Y):=\bigg((X,Y) S\begin{pmatrix} X\\ Y\end{pmatrix}\bigg)^r\cdot (\det S)^{-(({2r+k)}/{2})}\quad\text{where $S=S_E$ in (8.2.11)}. \end{equation}

Then for $\Phi \in S_\varrho (\Gamma _0(N))$, the scalar-valued $(S, \Lambda, \psi )$-Bessel period ${\mathcal {B}}_\Lambda (\Phi ; E)$ of $\Phi$ is defined by

(8.2.17)\begin{equation} {\mathcal{B}}_\Lambda(\Phi; E):=(B_\Lambda(\Phi;E), Q_{S,\varrho})_{2r}. \end{equation}

8.3 Explicit $L$-value formula in the vector-valued case

Let us state our explicit formula for holomorphic Siegel modular forms. In what follows, whenever we refer to a type of an admissible representation of $G$ over a non-archimedean local field, we use the standard classification due to Roberts and Schmidt [RS18].

Let $N$ be a squarefree integer. We say that a non-zero $\Phi \in S_\varrho (\Gamma _0(N))$ is a newform if:

1. (1) $\Phi \in S_\varrho (\Gamma _0(N))^{\text {new}}$;

2. (2) $\Phi$ is an eigenform for the local Hecke algebras for all primes $p$ not dividing $N$ and an eigenfunction of the local $U(p)$ operator (see Saha and Schmidt [SS13, 2.3]) for all primes dividing $N$;

3. (3) the representation $\pi (\Phi )$ of $G(\mathbb {A})$ is irreducible.

Then the following theorem is derived from Theorem 1.2 exactly as in Dickson, Pitale, Saha and Schmidt [DPSS20, Theorem 1.13] except that we need to compute local Bessel periods at the real place adapting to the vector valued case. We perform the computation of them in Appendix B.

Theorem 8.1 Let $N\ge 1$ be an odd squarefree integer. Let $\varrho =\varrho _\kappa$ where $\kappa =(2r+k,k)$ with $k\ge 2$. Let $\Phi$ be a non-CAP newform in $S_{\varrho }(\Gamma _0(N))$. Suppose that $({{D_E}/{p}})=-1$ for all primes $p$ dividing $N$. When $k=2$, suppose moreover that $\pi (\Phi )$ is tempered.

Then we have

(8.3.1)\begin{equation} \frac{|{\mathcal{B}}_\Lambda(\Phi;E)|^2}{\langle\Phi,\Phi\rangle_{\varrho}} =\frac{2^{4k+6r-c}}{D_E}\cdot \frac{L(1/ 2,\pi(\Phi) \times \mathcal{AI} (\Lambda))}{L(1,\pi(\Phi),\mathrm{Ad})}\cdot \prod_{p \,|\, N}J_p, \end{equation}

where $c=5$ if $\Phi$ is a Yoshida lift in the sense of Saha [Sah15, § 4] and $c=4$ otherwise. The quantities $J_p$ for $p$ dividing $N$ are given by

\[ J_p=(1+p^{-2})(1+p^{-1})\times \begin{cases} 1 & \text{if $\pi(\Phi)_p$ is of type $\mathrm{IIIa}$}; \\ 2 & \text{if $\pi(\Phi)_p$ is of type $\mathrm{VIb}$};\\ 0 & \text{otherwise.} \end{cases} \]

Remark 8.3 When $k\ge 3$, $\pi (\Phi )$ is tempered by Proposition 8.1.

Remark 8.4 Since $\mathcal {B}(\Phi ;E)=2^{k}D_E^{-({k}/{2})} \cdot B(\Phi ;E)$ when $r=0$, (1.8.2) follows from (8.3.1) by putting $N=1$ and $r=0$.

Remark 8.5 In the statement of the theorem, we used the notion of Yoshida lifts in the sense of Saha [Sah15]. Though it is necessary to extend the arguments concerning Yoshida lifts in [Sah15, § 4] in the scalar-valued case to the vector-valued case to be rigorous, we omit it here since it is straightforward. We also mention that the arguments in [Sah15, 4.4] now work unconditionally since the classification theory in Arthur [Art13] is complete for $\mathbb {G}=\mathrm {PGSp}_2\simeq \mathrm {SO}(3,2)$.

Remark 8.6 Recall that the $L$-functions in (8.3.1) are complete $L$-functions. We may rewrite the explicit formula in terms of the finite parts of the $L$-functions by observing that the relevant archimedean $L$-factors are given by

\[ L(1/ 2,\pi(\Phi)_\infty \times \mathcal{AI} (\Lambda)_\infty) =2^4(2\pi)^{-2(k+r)}\Gamma(k+r-1)^2\Gamma(r+1)^2 \]

and

\begin{align*} L(1,\pi(\Phi)_\infty,\mathrm{Ad}) &= 2^6(2\pi)^{-(4k+6r+1)}\Gamma(k+2r)\Gamma(k-1)\Gamma(2r+2)\Gamma(2k+2r-2), \end{align*}

respectively.

Remark 8.7 Let us consider the case when $D$ is a quaternion algebra over $\mathbb {Q}$ which is split at the real place, i.e. $D(\mathbb {R}) \simeq \mathrm {Mat}_{2 \times 2}(\mathbb {R})$. Assuming that the endoscopic classification holds for $\mathbb {G}_D=G_D/ Z_D$, we may apply Theorem 1.2 to holomorphic modular forms on $\mathbb {G}_D(\mathbb {A})$. In this case, Hsieh and Yamana [HY24] compute local Bessel periods and show an explicit formula for Bessel periods such as (8.3.1) for scalar-valued holomorphic modular forms, including the case when $G_D =G$ and $N$ is an even squarefree integer. Meanwhile, we shall maintain $N$ to be odd in Theorem 8.1, since our computation of the local Bessel period at the real place in the vector-valued case in Appendix B is performed under the assumption that $N$ is odd.

As we noted in Remark 1.5, after the submission of this paper, Ishimoto [Ish24] showed the endoscopic classification of $\mathrm {SO}(4,1)$ for generic Arthur parameters. Therefore, we may apply our theorem to the case of $\Bbb G_D \simeq \mathrm {SO}(4,1)$.

Remark 8.8 A global explicit formula such as (8.3.1) is obtained in a certain non-squarefree level case by Pitale, Saha and Schmidt [PSS23, Theorem 4.8].