Twisted GGP problems and conjectures

Abstract

In a series of three earlier papers, we considered a family of restriction problems for classical groups (over local and global fields) and proposed precise answers to these problems using the local and global Langlands correspondence. These restriction problems were formulated in terms of a pair $W \subset V$ of orthogonal, Hermitian, symplectic, or skew-Hermitian spaces. In this paper, we consider a twisted variant of these conjectures in one particular case: that of a pair of skew-Hermitian spaces $W = V$.

1. Introduction

In [GGP12b], we considered a family of restriction problems for classical groups (over local and global fields) and proposed precise answers to these problems using the local and global Langlands correspondence. These restriction problems were formulated in terms of a pair $W \subset V$ of orthogonal, Hermitian, symplectic, or skew-Hermitian spaces. In this paper, we consider a twisted variant of these conjectures in one particular case: that of a pair of skew-Hermitian spaces $W = V$.

Let $F$ be a non-archimedean local field and let $E$ be a separable quadratic algebra over $F$ with $\sigma \in {\rm Gal}(E/F)$, the non-trivial element of the Galois group. Let $V$ be a non-degenerate skew-Hermitian space of dimension $n$ over $E$, with pairing $\langle v,w \rangle$. We may choose an orthogonal basis $\{v_1,v_2,\ldots, v_n\}$ of $V$ over $E$ and define the determinant

\begin{align*} \det(V) = \prod_i \langle v_i,v_i\rangle, \end{align*}

in which each term $\langle v_i,v_i \rangle$ lies in $E_0^{\times }$, where $E_0$ is the $F$-subspace of $E$ consisting of elements of trace $0$ and $E_0^{\times } = E_0 \smallsetminus \{ 0\}$. Since the product of two elements in $E_0^{\times }$ lies in $F^{\times }$, the determinant lies in $E_0^{\times }$ when $n$ is odd and in $F^{\times }$ when $n$ is even. Both $E_0^{\times }$ and $F^{\times }$ are principal homogeneous spaces for the group $F^{\times }$, and the orbit spaces $E_0^{\times }/NE^{\times }$ and $F^{\times }/NE^{\times }$ have cardinality $2$ if $E$ is a field, and have cardinality 1 otherwise. The determinant, as an element of one of these orbit spaces $E_0^{\times }/NE^{\times }$ or $F^{\times }/NE^{\times }$, is independent of the choice of an orthogonal basis, and gives a complete isomorphism invariant of the skew-Hermitian space $V$ over $E$.

The isometry group ${\rm U}(V)$ has associated to it the Weil representation $\omega _{V,\psi, \mu }$ (see [GGP12b, pp. 47–50]). If $E$ is a field, then this complex representation of ${\rm U}(V)$ depends on a non-trivial additive character $\psi$ of $F$ and a conjugate-symplectic character $\mu$ of $E^{\times }$ (i.e. the restriction of $\mu$ to $F^{\times }$ is the quadratic character $\omega _{E/F}$ associated to $E/F$ by the local class field theory, so that $\omega _{E/F}: F^{\times }/ N(E^{\times }) \cong \{ \pm 1 \}$). For an irreducible representation $\pi _1 \otimes \pi _2$ of ${\rm U}(V) \times {\rm U}(V)$ with a generic $L$-parameter, we had considered the problem of determining

\begin{align*} \dim {\rm Hom}_{{\rm U}(V)}(\pi_1 \otimes \pi_2, \omega_{V, \psi, \mu}), \end{align*}

in [GGP12b]. It is known by the work [Sun12] that this dimension is $0$ or $1$, and the conjecture in [GGP12b] (proved in [GI16]) determines precisely when this dimension is equal to $1$.

If $E = F \times F$, ${\rm U}(V) \cong {\rm GL}_n(F)$, the Weil representation $\omega _{V,\psi, \mu }$ could be taken to be ${\mathcal {S}}(F^n)$ with the natural action of ${\rm GL}_n(F)$ on it, and the resulting Hom space ${\rm Hom}_{{\rm GL}_n(F)}(\pi _1 \otimes \pi _2, {\mathcal {S}}(F^n) )$ is the one which intervenes in the local Rankin–Selberg integral for ${\rm GL}_n(F) \times {\rm GL}_n(F)$.

Here is the simplest twisted variant of the above question that we consider in this paper. Instead of considering ${\rm U}(V)$ as a subgroup of ${\rm U}(V)(F \times F) = {\rm U}(V) \times {\rm U}(V)$, we consider it as a subgroup of ${\rm U}(V)(E) \cong {\rm GL}_n(E)$. For an irreducible generic representation $\Pi$ of ${\rm GL}_n(E)$, we consider the problem of determining

\begin{align*} \dim {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V, \psi, \mu}). \end{align*}

We conjecture that this dimension is equal to $1$ for a unique (up to isomorphism) skew-Hermitian space $V$ of dimension $n$ over $E$, whose determinant is related to a local epsilon factor that we describe now.

Let $M$ be the Langlands parameter of $\Pi$, thus $M$ is an $n$-dimensional representation of the Weil–Deligne group $WD_E$ of $E$. Associated to $M$, let ${}^\sigma \!M ^\vee$ be the conjugate-dual representation of $WD_E$, so that $M \otimes {}^\sigma \!M ^\vee$ is a conjugate-orthogonal representation of $WD_E$ of dimension $n^2$. Since $\mu |_{F^\times } = \omega _{E/F}$, $\mu$ is a conjugate-symplectic character of $E^{\times }$, and hence $M \otimes {}^\sigma \!M ^\vee \otimes \mu ^{-1}$ is a conjugate-symplectic representation of $WD_E$. In this paper, we conjecture that the skew-Hermitian space $V$ for which ${\rm Hom}_{{\rm U}(V)}(\Pi, \omega _{V, \psi, \mu }) \not = 0$ is determined by the identity

\begin{align*} \mu(\det(V)) = \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \det(M)(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

where $\psi _E$ is the additive character of $E$ obtained by composing $\psi$ with the trace from $E$ to $F$. For the other skew-Hermitian space $V'$ of rank $n$ over $E$, we conjecture that

\begin{align*} {\rm Hom}_{{\rm U}(V')}(\Pi, \omega_{V', \psi, \mu}) = 0. \end{align*}

We note that when $n$ is even so that $\det (V) \in F^\times$, $\mu (\det (V)) = \omega _{E/F} (\det (V))= \pm 1$, and $\mu (\det (V)) = +1$ if and only if the group ${\rm U}(V)$ is quasi-split. When $n$ is odd, the group ${\rm U}(V)$ is quasi-split for both of the skew-Hermitian spaces, and $\mu (\det (V))$ is a square root of $\omega _{E/F}(-1)$. Likewise, the local root number $\epsilon (1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu ^{-1}, \psi _E)$ is equal to $\pm 1$ when $n$ is even and is a square root of $\omega _{E/F}(-1)$ if $n$ is odd.

A related problem that has been studied in the literature is the determination of

\begin{align*} \dim {\rm Hom}_{{\rm U}(V)}( \Pi, \mathbb{C}). \end{align*}

The third author has proposed precise conjectures about this dimension [Pra20]. Here, we have replaced the trivial representation of ${\rm U}(V)$ by a Weil representation, which lies in a one-parameter family (indexed by the characters of $E^1$) of the next smallest representations of ${\rm U}(V)$. In retrospect, this appears quite natural and is simpler than this related problem considered in [Pra20]. It is also simpler than our original conjecture in the skew-Hermitian case, where we considered ${\rm U}(V)$ as a subgroup of ${\rm U}(V)(F \times F) = {\rm U}(V) \times {\rm U}(V)$, whereas ${\rm U}(V)(E)={\rm GL}_n(E)$ is a simpler group, whose $L$-packets are singletons. Note also that for ${\rm Hom}_{{\rm U}(V)}(\Pi, \omega _{V, \psi, \mu })$, we consider $\epsilon$ and $L$-function at $1/2$ of $M \otimes {}^\sigma \!M ^\vee \otimes \mu ^{-1}$ whereas for ${\rm Hom}_{{\rm U}(V)}( \Pi, \mathbb {C})$, one considers the pole at $s=1$ of $M \otimes {}^\sigma \!M ^\vee$.

The astute reader can no doubt guess by now the general twisted variant of the GGP conjecture we have in mind. Beyond the case of ${\rm U}(V)$ as a subgroup of ${\rm U}(V)(E)$ and ${\rm U}(V)(F \times F)$, we could choose a different quadratic extension $K$ of $F$ and consider ${\rm U}(V)$ as a subgroup of ${\rm U}(V)(K)$, which is the isometry group of the skew-Hermitian space $V \otimes _E L$, with $L = E \otimes K$. Indeed, one could consider an arbitrary pair of étale quadratic $F$-algebras $(E,K)$ and formulate a corresponding branching problem. The various possibilities are given in the following table.

Remark 1.1 We remark that in the case when $E=F \times F$, and $K$ is a separable quadratic extension of $F$ (corresponding to the first row of the above table), we would be asserting that for any irreducible admissible generic representation $\pi$ of ${\rm GL}_n(K)$, and for $\omega$ the Weil representation of ${\rm GL}_n(F)$ realized on the Schwartz space ${\mathcal {S}}(F^n)$, we have

\begin{align*} {\rm Hom}_{{\rm GL}_n(F)}[ \pi \otimes \omega, \mathbb{C}] = \mathbb{C}. \end{align*}

The assertion on dimension of the Hom space being $\leq 1$ is part of Theorem B of [Sun12], and that it is non-zero is the conclusion of the Rankin–Selberg theory.

The last case in the table above, when $E \ne K$ are two distinct quadratic fields, is the most complex and is discussed in § 8. To provide some evidence for our conjectures, we prove them when $n= \dim V \leq 2$ (see §§ 3 and 9), as well as for unitary principal series representations for general $n$ (see §§ 4, 5, and 10, especially Corollary 5.3 and Theorem 10.7). Indeed, when $E = K$, we reduce the conjecture for tempered representations to the case of essentially discrete-series representations of ${\rm GL}(V)$ (in Corollary 5.2), and further to the case of supercuspidal representations under a certain hypothesis (in Theorem 5.4). In particular, this allows us to prove the conjecture for the Steinberg representation (in Corollary 5.8). As a supplementary result, we show the vanishing of the corresponding higher Ext groups ${\rm Ext}^i$ ($i \geq 1$) for tempered representations (in Theorem 5.9).

The work which we needed to do in this paper with the Mackey theory allowed us to deal with certain non-tempered representations too, leading us naturally to the non-tempered analog of the GGP conjectures [GGP20] in the twisted setting. In considering this twisted case, we realized that our original conjectures for non-tempered representations, where we introduced the concept of relevant parameters, needed to be clarified in some cases. This is also done in § 7.

We also consider the twisted period problems over global fields. As in the GGP conjectures, one expects that the non-vanishing of the global period integral here too is equivalent to the non-vanishing of a corresponding central $L$-value, in the absence of local obstructions. For example, when $E = K$, the relevant central $L$-value is $L(1/2, M \times {}^\sigma \!M ^\vee \times \mu ^{-1})$. One can also formulate a refined conjecture in the style of Ichino–Ikeda, which gives a precise formula relating the global period integral to the product of the above central $L$-value and certain canonical local period integrals. In the global context, it is interesting to note that when $E \ne K$, all possible local scenarios given in the above table will arise. Hence, one of our goals in this paper is to give a uniform formulation of the local conjectures which can be specialized to all the local scenarios in the table.

With the twisted GGP problems and conjectures formulated, one can ask if all the previous work that has been done for the GGP conjectures can be adapted to this twisted setting. These include Waldspurger's and Beuzart-Plessis's integral formulae for the branching multiplicity and comparison of Jacquet–Rallis relative trace formulae, which in the skew-Hermitian case is due to Liu [Liu14] and Xue [Xue14, Xue16]. To this end, we remark that an integral formula for the branching multiplicity is being developed in the thesis work of Nhat Hoang Le (a student of the first author), whereas a relative trace formula approach is being pursued in the thesis work of Danielle Wang (a student of Wei Zhang at MIT).

2. When $E = K$ is a field

In this section, we consider the simpler case $E = K$, which was briefly discussed in the introduction. We formulate our conjectures more formally here, in both the local and global setting.

2.1 Local case

We assume first that $F$ is a local field and $E/F$ is a separable quadratic field extension. We let $E_0$ denote the $F$-subspace of trace $0$ elements in $E$ and let $E_1 \subset E^{\times }$ denote the subgroup of norm 1 elements. Fix a non-trivial additive character $\psi$ of $F$ and let $\sigma \in {\rm Gal}(E/F)$ be the non-trivial automorphism of $E/F$.

For a skew-Hermitian space $V$ over $E$ of dimension $n$, we recall that

\begin{align*} \det(V) \in \begin{cases} F^{\times}/ N_{E/F}(E^{\times}), & \text{if $n$ is even;} \\ E_0^{\times}/N_{E/F}(E^{\times}), & \text{if $n$ is odd.} \end{cases} \end{align*}

If $F$ is non-archimedean, there are precisely two skew-Hermitian spaces of dimension $n$, distinguished by their determinants. When $F$ is archimedean, there are many more skew-Hermitian spaces, distinguished by their signatures.

Without loss of generality, we may assume that all these skew-Hermitian spaces (of a given dimension) have the same underlying vector space $V$ over $E$, equipped with non-isomorphic skew-Hermitian forms. Thus the unitary groups ${\rm U}(V) \subset {\rm GL}(V)={\rm Aut}_E(V) = {\rm GL}_n(E)$ are all subgroups of a fixed ambient group ${\rm GL}_n(E)$.

For each skew-Hermitian space $V$ over $E$ and a conjugate-symplectic character $\mu$ of $E^{\times }$, we have the associated Weil representation $\omega _{V, \psi, \mu }$ of ${\rm U}(V)$. Now for an irreducible representation $\Pi$ of ${\rm GL}(V) \cong {\rm GL}_n(E)$, we consider the Hom space

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V, \psi, \mu}). \end{align*}

We now state our main local conjecture in this case.

Conjecture 2.1 (i) For any $\Pi \in {\rm Irr}({\rm GL}(V))$,

\begin{align*} \dim {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V, \psi, \mu}) \leq 1. \end{align*}

(ii) If $\Pi \in {\rm Irr}({\rm GL}(V))$ is generic, then

\begin{align*} \sum_V \dim {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V,\psi,\mu}) = 1. \end{align*}

where the sum is over the equivalence classes of skew-Hermitian structures on $V$.

(iii) For generic $\Pi \in {\rm Irr}({\rm GL}(V))$, the unique skew-Hermitian space $V$ which gives a non-zero contribution to the above sum satisfies

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, \Pi \times {}^\sigma\Pi ^\vee {\times} \mu^{-1}, \psi_E) \cdot \omega_{\Pi}(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

where ${}^\sigma \Pi ^{\vee }$ is the conjugate-dual representation of $\Pi$ and $\omega _{\Pi }$ is the central character of $\Pi$.

As noted in the introduction, the ratio of the two sides of condition (iii) is a priori $\pm 1$. When $F$ is non-archimedean, condition (iii) in the conjecture uniquely determines the summand with non-zero contribution to the sum in condition (ii). When $F = \mathbb {R}$ and $E = \mathbb {C}$, one needs to be more specific about the $V$ which gives non-zero contribution. We shall consider this archimedean case in greater detail in § 6. Note that if we define the discriminant of $V$ by

\begin{align*} {\rm disc}(V) = (-1)^{n(n-1)/2} \cdot \det(V), \end{align*}

then the formula in condition (iii) can be expressed more succinctly as

\begin{align*} \mu({\rm disc}(V)) = \epsilon( 1/2, \Pi \times {}^\sigma \Pi^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\Pi}(-1)^n, \end{align*}

taking note of the fact that $\mu (-1) = \omega _{E/F}(-1)$. We shall provide some evidence for this conjecture in the next two sections, verifying it for $\dim V \leq 2$ and for unitary principal series representations of ${\rm GL}(V)$ for $V$ of arbitrary dimension over $E$.

In the above formulation, the conjecture does not require the local Langlands correspondence, as the local root number in condition (iii) can be interpreted as the Rankin–Selberg local root number defined by Jacquet, Piatetski-Shapiro, and Shalika [JPSS83].

Let $M$ denote the Langlands parameter of $\Pi$, so that $M$ is an $n$-dimensional representation of the Weil–Deligne group $WD_E$ of $E$ with $\det (M)$ corresponding to the central character $\omega _{\Pi }$ under the local class field theory. We have noted in the introduction that $M \otimes {}^\sigma \!M ^\vee \otimes \mu ^{-1}$ is a conjugate-symplectic representation of $WD_E$. Then Conjecture 2.1(iii) can be written as

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \det(M)(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}. \end{align*}

Note that, for $e \in E_0^{\times }$,

\begin{align*} \det(M \otimes {}^\sigma \!M ^\vee)(e) = \det(M)(e)^n / \det(M)(e^{\sigma})^n = \det(M)(-1)^n, \end{align*}

and

\begin{align*} \omega_{E/F}(-1) = \omega_{K/F}(e^2) = (e^2, e^2) \quad \text{(Hilbert symbol)}. \end{align*}

Hence, the above identity can be expressed as (for $E=K$)

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \det(M \otimes {}^\sigma \!M ^\vee)(e) \cdot \omega_{K/F}(e^2)^{n(n-1)/2}, \end{align*}

and it is this last statement that generalizes well when we deal with the general case (where $E \ne K$) later.

2.2 Global case

Consider now the case when $E/ F$ is a quadratic extension of global fields with adele rings $\mathbb {A}_E$ and $\mathbb {A}_F$. Fix a non-trivial additive character $\psi$ of $F \backslash \mathbb {A}_F$. We shall consider all skew-Hermitian structures on a vector space $V$ of dimension $n$ over $E$.

Let $\Pi \cong \bigotimes _v \Pi _v$ be a cuspidal automorphic representation of ${\rm GL}(V)(\mathbb {A}_F) = {\rm GL}(V\otimes _F\mathbb {A}_F) = {\rm GL}(V\otimes _E \mathbb {A}_E)$ so that $\Pi _v$ are generic representations for each place $v$ of $E$. For a conjugate-symplectic Hecke character $\mu$ of $\mathbb {A}_E^{\times }$, we may consider the automorphic Weil representation $\omega _{V,\psi, \mu }$ of ${\rm U}(V)(\mathbb {A}_F)$ (see [GGP12b]). Now we consider the global period integral

\begin{align*} \mathcal{P}_V : \Pi\otimes \overline{\omega_{V,\psi, \mu}} \longrightarrow \mathbb{C} \end{align*}

defined by

\begin{align*} \mathcal{P}_V(f, \phi) = \int_{[{\rm U}(V)]} f(g) \cdot \overline{\phi(g)} \, dg \quad \text{for $f \in \Pi$ and $\phi \in \omega_{V,\psi, \mu}$,} \end{align*}

where we have written $[{\rm U}(V)]$ for the automorphic quotient ${\rm U}(V)(F) \backslash {\rm U}(V)(\mathbb {A}_F)$ with $dg$ the Tamagawa measure on it.

Globally, we are interested in characterizing the non-vanishing of this period integral. Our global conjecture is the following.

Conjecture 2.2 In the above setting, in particular for $V$ a skew-Hermitian space over a global field $E$, the global period integral $\mathcal {P}_V$ is non-zero if and only if the following two conditions hold (denoting $V_v=V \otimes F_v$):

1. (a) for all places $v$ of $F$, ${\rm Hom}_{{\rm U}(V_v)}( \Pi _v, \omega _{V_v, \psi _v, \mu _v}) \ne 0$;

2. (b) $L(1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) \ne 0.$

Further, for a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$, if $L(1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) \ne 0$, then there exists a unique skew-Hermitian space $V$ of dimension $n$ over $E$ such that the global period integral $\mathcal {P}_V$ is non-zero.

Observe that if we are given a collection of local skew-Hermitian spaces $\{V_v\}$ for all places $v$ of $F$ (of a fixed dimension $n \geq 1$), then the adelic skew-Hermitian space $\bigotimes _v V_v$ is coherent over $F$, i.e. the family of local skew-Hermitian spaces $V_v$ comes from a global skew-Hermitian space $V$, if and only if

\begin{align*} \prod_v \mu_v\big( \det(V_v) \big) = 1. \end{align*}

Therefore, given a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$, if the local skew-Hermitian spaces $\{V_v\}$ are those for which ${\rm Hom}_{{\rm U}(V_v)}( \Pi _v, \omega _{V_v, \psi _v, \mu _v}) \ne 0$ for all places $v$ of $F$, then part (iii) of our local Conjecture 2.1 implies that this collection of local skew-Hermitian spaces $\{V_v\}$ is coherent over $F$ if and only if

\begin{align*} \epsilon(1/2, \Pi \times {}^\sigma\Pi^\vee \times \mu^{-1}) = 1. \end{align*}

Therefore, given a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$ for which the global period integral on $[U(V)]$ is non-zero (hence, ${\rm Hom}_{{\rm U}(V_v)}( \Pi _v, \omega _{V_v, \psi _v, \mu _v}) \ne 0$ for all places $v$ of $F$), then $\epsilon (1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) = 1$. Thus, a necessary condition for the non-vanishing of $L(1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1})$ is satisfied if the global period integral on $[U(V)]$ is non-zero. Conversely, given a cuspidal automorphic representation $\Pi$ of ${\rm GL}_n(\mathbb {A}_E)$ for which $L(1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) \not = 0$ and hence $\epsilon (1/2, \Pi \times {}^\sigma \Pi ^\vee \times \mu ^{-1}) = 1$, we have a global skew-Hermitian space $V$, unique up to isomorphism, for which Conjecture 2.2 implies non-vanishing of period integral on $[{\rm U}(V)]$.

2.3 A refined global conjecture

Not surprisingly, one expects to be able to refine the above global conjecture to a precise formula relating the global period integral to the central $L$-value.

For $\Pi \cong \bigotimes _v \Pi _v$, a cuspidal automorphic representation of ${\rm GL}(V)(\mathbb {A}_F) = {\rm GL}(V\otimes _F\mathbb {A}_F) = {\rm GL}(V\otimes _E \mathbb {A}_E)$, and $\omega _{V, \psi, \mu } \cong \bigotimes _v \omega _{V_v, \psi _v, \mu _v}$, the Weil representation of ${\rm U}(V)(\mathbb {A}_F)$, $f_v,f'_v \in \Pi _v$ and $\phi _v,\phi '_v \in \omega _{V_v, \psi _v, \mu _v}$, we may consider the following integral of matrix coefficients for each place $v$ of $F$:

(2.3)\begin{equation} \mathcal{I}_v ( f_v, f'_v, \phi_v, \phi'_v) := \int_{{\rm U}(V)(F_v)} \langle g_v \cdot f_v, f'_v \rangle \cdot \overline{ \langle g_v \cdot \phi_v, \phi'_v\rangle} \, dg_v. \end{equation}

As in [Xue16], it is not hard to see that if $\Pi _v$ is tempered, this integral is absolutely convergent, so that it defines a ${\rm U}(V_v) \times {\rm U}(V_v)$-equivariant linear functional

\begin{align*} \mathcal{I}_v : \Pi_v \otimes \overline{\Pi}_v \otimes \overline{\omega_{V_v, \psi_v, \mu_v}} \otimes \omega_{V_v, \psi_v,\mu_v} \longrightarrow \mathbb{C}. \end{align*}

Now one would like to:

• • show that $\mathcal {I}_v$ is non-zero if and only if ${\rm Hom}_{{\rm U}(V)(F_v)}( \Pi _v , \omega _{V_v, \psi _v,\mu _v}) \ne 0$;

• • compute this integral at almost all places $v$ of $F$ where all data involved are unramified.

Without having done this work, we may nonetheless venture a guess here, in analogy with the original GGP case [Xue16].

Conjecture 2.4 Suppose that:

• • $E_v/F_v$ is an unramified quadratic extension of residue characteristic not $2$ and $\psi _v$ has conductor $\mathcal {O}_{F_v}$;

• • $\mu _v$ is unramified;

• • $V_v$ contains a selfdual lattice $\Lambda _v$ whose stabilizer in ${\rm U}(V_v)$ is a hyperspecial maximal compact subgroup $K_v$, contained in $\tilde {K}_v = {\rm GL}(\Lambda _v) \subset {\rm GL}(V_v)$;

• • $dg_v$ is the Haar measure on ${\rm U}(V_v)$ which gives $K_v$ volume $1$;

• • $\Pi _v$ is $\tilde {K}_v$-unramified and $f_v = f'_v$ is a $\tilde {K}_v$-spherical vector of norm 1;

• • $\phi _v = \phi '_v$ is a $K_v$-spherical vector of norm 1 in the Weil representation $\omega _{V_v, \psi _v, \mu _v}$.

Then

\begin{align*} \mathcal{I}_v ( f_v, f'_v, \phi_v, \phi'_v) = \frac{L(1, M_{{\rm GL}(V_v)}^{\vee})}{L(1, M^{\vee}_{{\rm U}(V_v)})} \cdot \frac{ L(1/2, \Pi_v \times {}^\sigma\Pi^\vee_v \times \mu_v^{-1})}{L(1, \Pi_v, {\rm Ad})}, \end{align*}

where

\begin{align*} L(1, M_{{\rm GL}(V_v)}^{\vee}) = \prod_{k=1}^n \zeta_{E_v}(k) \quad \text{and} \quad L(1, M^{\vee}_{{\rm U}(V_v)}) = \prod_{k=1}^n L(k, \omega_{E_v/F_v}^k) \end{align*}

are the values at $s=1$ of the $L$-functions of the dual motives of ${\rm GL}(V)$ and ${\rm U}(V)$, respectively. (One may observe that the expression for $\mathcal {I}_v ( f_v, f'_v, \phi _v, \phi '_v)$ given above implies, in particular, that it is non-zero.)

Given this, it is natural to define a normalized local period integral:

(2.5)\begin{equation} \mathcal{I}_v^{\#} = \frac{L(1, M^{\vee}_{{\rm U}(V_v)})}{L(1, M_{{\rm GL}(V_v)}^{\vee})} \cdot \frac{L(1, \Pi_v, {\rm Ad})}{L(1/2, \Pi_v \times {}^\sigma\!\Pi^\vee_v \times \mu_v^{-1})} \cdot \mathcal{I}_v. \end{equation}

We also note that if $E_v = F_v \times F_v$, the analog of the above conjecture holds, and has already been considered in the original formulation of the refined GGP conjecture for skew-Hermitian spaces in [GGP12b].

Coming back to the global setting, for each of the groups ${\rm GL}(V)$ or ${\rm U}(V)$, we will fix a decomposition of the Tamagawa measures $dg = \prod _v dg_v$, so that for almost all $v$, the local Haar measures $dg_v$ give a hyperspecial maximal compact subgroup volume $1$. We will also fix a decomposition of the global Petersson inner product (defined by integrating over ${\rm GL}_n(E) \backslash {\rm GL}_n({\mathbb{A}}_E)^1$, where ${\rm GL}_n({\mathbb{A}}_E)^1 \,{:=}\, \{ g \in {\rm GL}_n({\mathbb{A}}_E)\ {\rm with}\ |{\rm det}(g)| =1 \} $) as a product of local pairings:

(2.6)\begin{equation} \langle- , - \rangle_{{\rm Pet}} = \prod_v \langle-, - \rangle_v, \end{equation}

and use these $dg_v$ and $\langle -,-\rangle _v$ in the definition of the local period integrals $\mathcal {I}_v$ introduced above. We can now state the following.

Conjecture 2.7 Given a (tempered) cuspidal automorphic representation $\Pi$ of ${\rm GL}(V)$,

\begin{align*} \mathcal{P} \otimes \overline{\mathcal{P}}= \frac{ L(1/2, \Pi \times {}^\sigma\Pi^\vee \times \mu^{-1})}{L(1, M_{{\rm U}(V)}^{\vee})} \cdot \bigg( \frac{L(s, M_{{\rm GL}(V)}^{\vee})}{L(s, \Pi, {\rm Ad})} \bigg)\bigg|_{s=1} \cdot \prod_v \mathcal{I}_v^{\#} . \end{align*}

as linear functionals on $\Pi \otimes \overline {\Pi } \otimes \overline {\omega _{V,\psi, \mu }} \otimes \omega _{V, \psi, \mu }$.

Here, note that $L(s, M_{{\rm GL}(V)}^{\vee })$ and $L(s, \Pi, {\rm Ad})$ both have a simple pole at $s=1$, so that their ratio is holomorphic and non-zero at $s=1$.

2.4 Finite fields

We conclude this section by highlighting the restriction problem for skew-Hermitian spaces over a finite field $F = \mathbb {F}_q$. In the finite field setting, only the case $E = K$ can occur. In this setting, a naive first guess is that for any irreducible generic representation $\Pi$ of ${\rm GL}_n(\mathbb {F}_{q^2})$,

\begin{align*} \dim {\rm Hom}_{{\rm U}_n(\mathbb{F}_q)} (\Pi, \omega) =1, \end{align*}

where $\omega$ is the Weil representation of ${\rm Sp}_{2n}(\mathbb {F}_q)$, restricted to the subgroup ${\rm U}_n(\mathbb {F}_q)$. However, an examination of the case $n=1$ shows that this cannot literally be the case because $\dim \omega = q$ but ${\rm U}_1(\mathbb {F}_q)$ has $q+1$ characters. Indeed, the unique non-trivial quadratic character of ${\rm U}_1(\mathbb {F}_q)$ is missing from $\omega$. Moreover, experience with the usual GGP problem over finite fields shows that the above branching multiplicity can be larger than 1. Nonetheless, the naive expectation should be generically true for cuspidal Deligne–Lusztig representations and it is an interesting question to quantify the extent of its failure.

Over finite fields, we can also consider this restriction problem for symplectic groups. For any irreducible generic representation $\Pi$ of ${\rm Sp}_{2n}(\mathbb {F}_{q^2})$, one would thus like to determine

\begin{align*} \dim {\rm Hom}_{{\rm Sp}_{2n}(\mathbb{F}_q)}(\Pi, \omega) . \end{align*}

It is curious that since the two fold cover of ${\rm Sp}_{2n}(E)$ splits over ${\rm Sp}_{2n}(F)$, there is no analogous problem for non-archimedean local fields. Perhaps, one could go to four fold cover of ${\rm Sp}_{2n}(E)$ (if the 4th roots of unity are there in $E$) to study the analogous branching problem?

A first study of these branching problems over finite fields has been conducted by Nhat Hoang Le.

3. Evidence in low rank

In this section, we provide some evidence towards Conjecture 2.1 when $n = \dim V \leq 2$.

3.1 Rank-one case

We begin by examining the case when $\dim V =1$, so that ${\rm GL}(V) = E^{\times } \supset {\rm U}(V) = E_1$, where $E_1$ denotes the subgroup of norm one elements. Given a character $\chi$ of $E^{\times }$, we are thus interested in understanding ${\rm Hom}_{E_1}( \chi, \omega _{V,\psi, \mu })$. This is addressed by a theorem of Moen [Moe87] and Rogawski [Rog92].

Theorem 3.1 If $\chi$ is a character of $E^{\times }$, then

\begin{align*} \dim {\rm Hom}_{E_1}( \chi, \omega_{V,\psi, \mu}) \leq 1 \end{align*}

and equality holds if and only if

\begin{align*} \mu(\det(V)) = \chi(-1) \cdot \epsilon(1/2, \chi^{\sigma}/\chi \cdot \mu^{-1}, \psi_E). \end{align*}

This is precisely what Conjecture 2.1 asserts in the case $\dim V =1$.

3.2 Rank-two case

Suppose now that $\dim V = 2$. Skew-Hermitian spaces of rank two can be described using quaternion $F$-algebras, as we have exploited in [GGP12a]. More precisely, for a quaternion $F$-algebra $B$, fix an $F$-algebra embedding $i: E \hookrightarrow B$ and write $B = E \oplus E \cdot x$ where $x$ is an element of $B$ such that $x e x^{-1} = e^{\sigma }$. Thus, $B$ is a two-dimensional $E$-vector space (by left multiplication), and we may identify ${\rm GL}_E(B)$ with ${\rm GL}_2(E)$ with respect to the basis $\{1, x\}$.

Now fix a trace 0 element $\delta \in E^{\times }$ and set

\begin{align*} \langle b_1, b_2 \rangle = \delta \cdot (\text{projection of $b_1 \cdot \bar{b}_2$ onto $E$}). \end{align*}

Then $\langle -, -\rangle$ is a skew-Hermitian form on $B$; we shall denote this skew-Hermitian space by $V_B$. The isomorphism class of $V_B$ is independent of $x,\delta$, and $V_B$ is split if and only if $B$ is split.

The unitary similitude group ${\rm GU}(V_B) \subset {\rm GL}(V_B) = {\rm GL}_2(E)$ can be described by the isomorphism

\begin{align*} \iota: (B^{\times} \times E^{\times}) / \Delta F^{\times} \stackrel{\cong} {\longrightarrow} {\rm GU}(V_B) \subset {\rm GL}(V_B) \end{align*}

given by sending $(b,e) \in B^{\times } \times E^{\times }$ to the element of ${\rm GL}(V_B)$ whose action on $B$ is

\begin{align*} (b,e): y \mapsto e \cdot y \cdot b^{-1}. \end{align*}

The similitude character is

\begin{align*} {\rm sim}(b, e) = N_{E/F}(e) \cdot N_B(b)^{-1}. \end{align*}

Hence, the unitary group is

\begin{align*} {\rm U}(V_B) \cong \{ (b, e) \in (B^{\times} \times E^{\times}) / \Delta F^{\times} = {\rm GU}(V_B): N_{E/F}(e) = N_B(b) \}. \end{align*}

This is contained in the subgroup

\begin{align*} {\rm GU}(V_B)^+ \cong \{ (b,e) \in (B^{\times} \times E^{\times}) / \Delta F^{\times} ={\rm GU}(V_B): N_B(b) \in N_{E/F}(E^{\times}) \}, \end{align*}

which has index 2 in ${\rm GU}(V_B)$. Moreover, if $Z = E^{\times }$ denotes the center of ${\rm GL}(V_B)$, then

\begin{align*} {\rm GU}(V_B)^+ = Z \cdot {\rm U}(V_B). \end{align*}

Thus, when working with irreducible representations of ${\rm U}(V_B)$, there is no difference in working with ${\rm GU}(V_B)^+$ instead.

Let us explicate the Weil representation of ${\rm U}(V_B)$ in this framework. The Weil representation $\omega _{\psi,\mu, B}$ is reducible but admits a central character decomposition:

\begin{align*} \omega_{\psi, \mu, B} = \bigoplus_{\lambda} \omega_{\psi,\mu, B}[\lambda], \end{align*}

where the sum runs over the characters of $Z({\rm U}(V_B))= E_1$ and each summand is irreducible or $0$. We can describe $\omega _{\psi,\mu, B}[\lambda ]$ in terms of the description of ${\rm U}(V_B)$ given above. More precisely, suppose that $\lambda = \chi |_{E_1}$ for a character $\chi$ of $E^{\times }$. Consider the $L$-parameter

\begin{align*} N = {\rm Ind}_{W_E}^{W_F} (\mu \cdot \chi^{-1}) \quad \text{ of ${\rm GL}_2(F)$,} \end{align*}

and let $\Sigma _{B,N}$ be the associated representation of $B^{\times }$. This gives a representation

\begin{align*} \Sigma_{B,N} \boxtimes \chi \quad \text{ of $B^{\times}\times E^{\times}$}, \end{align*}

which is trivial on $\Delta F^{\times }$, i.e. a representation of ${\rm GU}(V_B)$. This representation of ${\rm GU}(V_B)$ decomposes into the sum of two irreducible summands when restricted to ${\rm GU}(V_B)^+$. One of these summands is the representation $\omega _{\psi,\mu, B} [\chi |_{E_1}]$ whereas the other is $\omega _{\psi ',\mu, B} [\chi |_{E_1}]$, with $\psi '$ in a different $N(E^{\times })$-orbit as $\psi$.

Now suppose that $\Pi$ is an irreducible generic representation of ${\rm U}(V_B \otimes _F E) \cong {\rm GL}(V_B)$ with $L$-parameter $M$. The embedding ${\rm U}(V_B) \hookrightarrow {\rm U}(V_B \otimes _F E)$ is the natural embedding ${\rm U}(V_B) \subset {\rm GL}(V_B)$. Pulling $\Pi$ back via $\iota$, and with $\chi := \omega _{\Pi }$, we see that

\begin{align*} {\rm Hom}_{{\rm U}(V_B)}(\Pi, \omega_{\psi,\mu, B}) &= {\rm Hom}_{{\rm U}(V_B)}(\Pi, \omega_{\psi,\mu, B}[\chi|_{E_1}]) \\ &\cong {\rm Hom}_{(B^{\times})^+}( \iota^*(\Pi), \omega_{\psi,\mu, B}[\chi|_{E_1}]) \\ &\cong {\rm Hom}_{B^{\times}}( \iota^*(\Pi), \Sigma_{B,N}). \end{align*}

Now it is important to note that the embedding $\iota : B^{\times} \hookrightarrow {\rm GL}(V_B) = {\rm GL}_2(E)$ is not the natural embedding $B^{\times } \hookrightarrow (B \otimes _F E)^{\times } \cong {\rm GL}_2(E)$, but rather differs from it by the outer automorphism $b \mapsto \bar {b}^{-1}$. Indeed, $\iota$ is the inverse on the central $F^{\times }$. Taking this into account, we see that the last Hom space above is the space

\begin{align*} {\rm Hom}_{B^{\times}}(\Pi^{\vee} \otimes \Sigma_{B,N}^{\vee}, \mathbb{C}) \end{align*}

of twisted trilinear forms, where $B^{\times } \hookrightarrow (B \otimes _F E)^{\times } \cong {\rm GL}_2(E)$, with the last isomorphism induced by an $E$-algebra isomorphism $B \otimes _F E \cong {\rm M}_2(E)$.

By a result of the third author [Pra92], one has

\begin{align*} \dim {\rm Hom}_{B^{\times}}(\Pi^{\vee} \otimes \Sigma_{B,N}^{\vee}, \mathbb{C}) \leq 1 \end{align*}

with equality if and only if

\begin{align*} \epsilon(1/2, {\rm As}^+(M^{\vee}) \otimes N^{\vee}, \psi_E) \cdot \omega_{E/F}(-1) = \mu(\det(V_B)), \end{align*}

where $As^+$ is the Asai lift of $M$ from $E$ to $F$. We refer the reader to § 8.4 for the definition and properties of $As^+$. Now let us explicate the local root number:

\begin{align*} \epsilon(1/2, {\rm As}^+(M^{\vee}) \otimes N^{\vee}, \psi_E) &= \epsilon(1/2, {\rm As}^+(M^{\vee}) \otimes {\rm Ind}_E^F (\mu^{-1} \cdot \chi),\psi) \\ &= \epsilon(1/2, {\rm Ind}_E^F ( M^{\vee} \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1} \otimes \chi), \psi) \\ &= \epsilon(1/2, {\rm Ind}_E^F ( M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}),\psi) \\ & = \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E), \end{align*}

where in the second last equality, we have noted that $\chi = \omega _{\Pi } = \det M$, so that $M^{\vee } \otimes \chi \cong M$ (since $\dim M = 2$), and in the last equality, we have used the fact that epsilon factors are inductive in dimension zero together with the fact that $\dim (M \otimes {}^\sigma \!M ^\vee ) =4$.

To conclude, we have shown the following.

Proposition 3.2 For an irreducible generic representation $\Pi$ of ${\rm U}(V_B \otimes _F E) \cong {\rm GL}(V_B)$, with $L$-parameter $M$ (a two-dimensional representation of $WD_E$),

\begin{align*} {\rm Hom}_{{\rm U}(V_B)}(\Pi, \omega_{\psi,\mu, B}) \ne 0 \iff \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \otimes \mu^{-1}, \psi_E) \cdot \omega_{E/F}(-1) = \mu(\det(V_B)). \end{align*}

This is precisely what Conjecture 2.1 says in the case $n =2$.

3.3 Global conjecture: rank-one case

Finally, we can also verify the global Conjecture 2.7 when $\dim _E V = 1$. Let $\chi$ be a Hecke character of ${\rm GL}(V)(\mathbb {A}_E) = \mathbb {A}_E^{\times }$, so that we are considering the global period integral

\begin{align*} \mathcal{P}: \mathbb{C} \chi \otimes \overline{\omega_{V,\psi,\mu}} \longrightarrow \mathbb{C} \end{align*}

defined by

\begin{align*} \mathcal{P}(\phi) = \int_{[E_1]} \chi(x) \cdot \overline{\phi(x)} \, {{d} x}. \end{align*}

Observe that this is simply the (conjugate of the) global theta lifting of $\chi$ for the dual pair

\begin{align*} {\rm U}_1 \times {\rm U}_1 = {\rm U}(V) \times {\rm U}(W), \end{align*}

evaluated at the identity element. Here, $V$ is equipped with its given skew Hermitian structure and $W$ is the rank-one Hermitian space $\langle 1 \rangle$. The non-vanishing of $\mathcal {P}$ is thus equivalent to the non-vanishing of the global theta lift $\Theta _{V,W,\psi,\mu }(\chi )$ of $\chi$. Moreover, when this global theta lift is non-zero, it is isomorphic to the representation $\chi$ of ${\rm U}(W) = E_1$. Then we have

\begin{align*} \mathcal{P}(\phi_1) \cdot \overline{\mathcal{P}(\phi_2)} \cdot \mu([E_1]) = \langle \Theta(\phi_2,\chi), \Theta(\phi_1,\chi) \rangle_{\rm Pet}, \end{align*}

where $\mu ([E_1]) =2$ is the Tamagawa measure of ${\rm U}(W)$. Now the Petersson inner product of the global theta lift on the right-hand side is computed by the Rallis inner product formula. This was first done by Tonghai Yang [Yan97] and a convenient reference is [Xue16, Theorem A.4.2]. One has

\begin{align*} \langle \Theta(\phi_2,\chi), \Theta(\phi_1,\chi) \rangle_{\rm Pet} = \frac{L(1/2, \chi^{\sigma} \chi^{-1} \cdot\mu^{-1})}{L(1, \omega_{E/F})} \cdot Z^*(\phi_2, \phi_1), \end{align*}

where

\begin{align*} Z^*(\phi_2,\phi_1)= \int^*_{\mathbb{A}_E^1} \overline{\langle g \phi_1,\phi_2\rangle}\cdot \langle g \chi, \chi \rangle_{{\rm U}(V), {\rm Pet}} \, dg \end{align*}

is the normalized global doubling zeta integral. Since the Tamagawa measure of ${\rm U}(V)$ is $2$, one has

\begin{align*} \langle g \cdot \chi, \chi \rangle_{{\rm U}(V), {\rm Pet}} = 2 \cdot \chi(g) \end{align*}

so that

\begin{align*} Z^*(\phi_1,\phi_2) = 2 \cdot \int^*_{\mathbb{A}_E^1} \overline{\langle g \phi_1,\phi_2\rangle}\cdot \chi(g) \, dg = \prod_v \mathcal{I}_v^{\#}(\chi,\chi, \phi_1,\phi_2), \end{align*}

where the local factors $\mathcal {I}_v^{\#}$ are as defined in (2.3) and (2.5). Hence, we conclude that

(3.3)\begin{equation} \mathcal{P}(\phi_1) \cdot \overline{\mathcal{P}(\phi_2)} = \frac{L(1/2, \chi^{\sigma} \chi^{-1} \cdot\mu^{-1})}{L(1, \omega_{E/F})} \cdot \prod_v \mathcal{I}_v^{\#}(\chi,\chi, \phi_1,\phi_2). \end{equation}

This is precisely what Conjecture 2.7 says.

For the case when $\dim _E V = 2$, Conjecture 2.7 should reduce to Ichino's formula [Ich08] relating the (twisted) triple product period integral and the (twisted) triple product $L$-value. We leave the verification of this to the interested reader.

4. Mackey theory: restriction of principal series

In this section, we apply Mackey theory to understand the branching of a principal series representation of ${\rm GL}(V)$ to the Weil representation of ${\rm U}(V)$ over a non-archimedean local field.

4.1 Principal series

Let $V$ be a vector space of dimension $n$ over $E$. For a partition $n = a+b,$ with $0< a \leq b \in \mathbb {Z}$, let

\begin{align*} V = V_a \oplus V_b \end{align*}

with $\dim V_a = a$ and $\dim V_b = b$. Consider the maximal parabolic subgroup

\begin{align*} P = P_{a,b} = M \cdot N \end{align*}

of ${\rm GL}(V)$ stabilizing $V_a$, with Levi factor

\begin{align*} M= {\rm GL}(V_a) \times {\rm GL}(V_b). \end{align*}

Let $\pi = \pi _1 \boxtimes \pi _2$ be a representation of ${\rm GL}(V_a)\times {\rm GL}(V_b)$ and consider the (normalized) parabolically induced representation

(4.1)\begin{equation} \pi = \pi_1 \times \pi_2 = {\rm Ind}_P^{{\rm GL}(V)} (\pi_1 \boxtimes \pi_2). \end{equation}

These are the principal series representations we will consider.

4.2 Skew-Hermitian structures

Recall that there are two inequivalent skew-Hermitian structures on $V$, distinguished by their determinants in $F^{\times }/NE^{\times }$ or $E_0^{\times }/ NE^{\times }$ (depending on whether $n = \dim V$ is even or odd). For such a class $\delta$, we let $V_{\delta }$ denote the skew-Hermitian structure on $V$ with determinant $\delta$, so that ${\rm U}(V_{\delta }) \subset {\rm GL}(V)$. We often drop $\delta$ from $V_\delta$ when a particular skew-Hermitian structure is fixed on $V$. We also let ${\rm rk}(V)$ denote the dimension of a maximal isotropic subspace of the skew-Hermitian space $V$; this is sometimes called the Witt index of $V$.

On the other hand, $V$ with a roman subscript, such as $V_a$, will denote either just a vector space over $E$ or a skew-Hermitian space over $E$ of dimension $a$.

4.3 Mackey theory

For a fixed skew-Hermitian space $V = V_{\delta }$, the goal of this section is to compute

\begin{align*} {\rm Hom}_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi,\mu}), \end{align*}

where $\omega _{V, \psi, \mu }$ denotes the Weil representation of ${\rm U}(V)$ associated to $(\psi,\mu )$. In fact, we will consider the more general

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi,\mu}). \end{align*}

This will be achieved by using Mackey theory, which requires the determination of the orbits of ${\rm U}(V)$ on the partial flag variety ${\rm GL}(V)/P$. In this analysis, each orbit gives rise to a certain induced representation of ${\rm U}(V)$ arising from the restriction of the inducing data to the stabilizer of a point in the orbit.

Thus, the representation $\pi = \pi _1 \times \pi _2$ when restricted to ${\rm U}(V)$ comes equipped with a certain finite filtration by ${\rm U}(V)$-modules in which the open orbits contribute as submodules, and the non-open orbits contribute as subquotients.

4.4 Orbits

The following lemma, the proof of which is omitted, summarizes the orbit structure of ${\rm U}(V)$ on ${\rm GL}(V) / P$ and is a direct consequence of Witt's theorem.

Lemma 4.2 The orbits of ${\rm U}(V)$ on ${\rm GL}(V) / P_{a,b}$ (with $0< a \leq b$) are represented by the isometry classes of $a$-dimensional $E$-subspaces $X \subset V$, which are themselves parameterized by the following two invariants:

1. (1) the dimension $d$ of the kernel of the skew-Hermitian form on $V$ restricted to $X$, i.e. $d=\dim (X \cap X^\perp )$, $d \leq \min \{a, {\rm rk}(V)\}$; and

2. (2) the non-degenerate skew-Hermitian form on $X/(X \cap X^\perp )$, which can be arbitrary.

In particular, with $E$ non-archimedean, one has the following.

• • For each integer

  \begin{align*} 0 \leq d \leq \min\{a, {\rm rk}(V)\}, \end{align*}
  there are two orbits $[X]$ of ${\rm U}(V)$ on ${\rm GL}(V) / P_{a,b}$, with $\dim (X \cap X^{\perp }) = d$, unless $d=a$ (i.e. when $X/(X \cap X^\perp ) = 0$), in which case there is only one.

• • The open orbits correspond to $d=0$, i.e. the isomorphism classes of the two non-degenerate skew-Hermitian subspaces of $V$ of dimension $a$.

• • There is a unique closed orbit which corresponds to $d = \min \{a, {\rm rk}(V)\} =a$, except when $n=2a$ and $V$ does not have an isotropic subspace of dimension $a$, in which case there are two closed orbits corresponding to $d=a-1$.

4.5 Stabilizers

Let $[X]$ be an ${\rm U}(V)$-orbit in ${\rm GL}(V)/P_{a,b}$, represented by an $E$-subspace $X \subset V$ of dimension $a$ with $\dim (X \cap X^\perp ) =d$. Let us first determine the stabilizer $S=S_X$ of $X$ in ${\rm U}(V)$.

Observe that $S_X$ preserves the flag

\begin{align*} 0 \subset X \cap X^\perp \subset X \subset ( X \cap X^\perp)^\perp = X+X^\perp \subset V, \end{align*}

and note that

\begin{align*} X/( X \cap X^\perp ) \subset (X+X^\perp) / ( X \cap X^\perp) =: V_{n-2d} \end{align*}

are non-degenerate skew-Hermitian spaces of dimension $a-d$ and $n-2d$, respectively. Hence, $S_X$ is contained in the maximal parabolic subgroup $Q_d$ of ${\rm U}(V)$ stabilizing the isotropic space $X \cap X^{\perp }$. The parabolic subgroup $Q_d =M_d \cdot N_d$ can be depicted in matrix form as

\begin{align*} Q_d = \Bigg(\begin{array}{@{}ccc@{}} {\rm GL}(X \cap X^{\perp}) & *_1 & *_d \\ 0 & {\rm U}(V_{n-2d}) & *_2 \\ 0 & 0 & * \end{array} \Bigg), \end{align*}

with Levi factor

\begin{align*} M_d = {\rm GL}(X \cap X^{\perp}) \times {\rm U}(V_{n-2d}) \end{align*}

and unipotent radical $N_d$. The center of $N_d$ is the subgroup $Z_d$ consisting of matrices with $*_1=*_2 =0$.

It follows that, as a subgroup of $Q_d$, $S_X$ has the form:

\begin{align*} S_X = \left(\begin{array}{@{}cccc@{}} g & *_{12} & *_{13} & *_4 \\ 0 & {\rm U}_{a-d} & 0 & *_{24} \\ 0 & 0 & {\rm U}_{b-d} & *_{34} \\ 0 & 0 & 0 & (g^*)^{-1} \\ \end{array} \right), \end{align*}

where:

• • $g \in {\rm GL}(X \cap X^{\perp }) \cong {\rm GL}_d(E)$;

• • the entries $*_{12}$ and $*_{34}$ are arbitrary matrices with entries in $E$ of appropriate sizes which determines $*_{24}$, $*_{13}$;

• • the entry $*_4$ is an arbitrary skew-Hermitian matrix of size $d \times d$.

Let us highlight certain natural subgroups or quotients of $S_X$.

• • The unipotent radical $N(S_X)$ of $S_X$ consists of those matrices which have the identity matrix on each diagonal block. Observe that $N(S_X)$ is, in fact, the unipotent radical $N_d$ of the maximal parabolic subgroup $Q_d$.

• • The center $Z(S_X)$ of $N(S_X)$ is the subgroup consisting of elements whose only non-zero entry in the upper triangular blocks is $*_4$, so that $Z(S_X) =Z_d$.

• • The Levi factor $S_X/ N(S_X)$ is isomorphic to

  \begin{align*} {\rm GL}(X \cap X^{\perp}) \times {\rm U}_{a-d} \times {\rm U}_{b-d}. \end{align*}

4.6 Modules

In what follows, we use ${\rm Ind}$ for the usual normalized induction, and ${\rm ind}$ for the usual normalized induction with compact support, whereas we will use $\mathfrak {I} nd$ and $\mathfrak {i} nd$ for the corresponding un-normalized induction. Thus, for example,

\begin{align*} \pi = \pi_1 \times \pi_2 = {\rm Ind}_P^{{\rm GL}(V)} (\pi_1 \otimes \pi_2) = \mathfrak{I} nd_P^{{\rm GL}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_P^{1/2}). \end{align*}

By Mackey theory, the restriction of the principal series representation $\pi = \pi _1 \times \pi _2$ to ${\rm U}(V)$ has a finite equivariant filtration indexed by the ${\rm U}(V)$-orbits given in Lemma 4.2. For each such ${\rm U}(V)$-orbit $[X]$, let $\pi _X$ denote the associated ${\rm U}(V)$-subquotient of $\pi$. The following proposition determines the representation $\pi _X$.

Proposition 4.3 For a ${\rm U}(V)$-orbit $[X]$ on ${\rm GL}(V)/P$, with $\dim X \cap X^{\perp } = d$ and stabilizer $S= S_X$, one has

\begin{align*} \pi_X \cong \mathfrak{i} nd_{S}^{{\rm U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_P^{1/2})|_{S} = {\rm ind}_{S}^{{\rm U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})|_{S}, \end{align*}

where we have written $\delta _{P/S} = \delta _P \delta _S^{-1}$.

We note that the representation $\pi _1 \otimes \pi _2 \otimes \delta _{P/S}^{1/2}$ is non-trivial on the unipotent radical $N(S)$ of $S$, but it is trivial on the center $Z(S)$ of $N(S)$.

4.7 Branching for $\pi _X$

We are now ready to consider the branching problem

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi, \mu}). \end{align*}

Since, as a ${\rm U}(V)$-module, $\pi _1 \times \pi _2$ has a finite filtration with subquotients $\pi _X$ as given in Proposition 4.3, it is natural to first consider

\begin{align*} {\rm Ext}^i_{{\rm U}(V)} (\pi_X, \omega_{V, \psi, \mu}). \end{align*}

The result of this key computation is given by the following proposition.

Proposition 4.4 For an orbit $[X]$ of ${\rm U}(V)$ on ${\rm GL}(V)/P$, with $\dim (X \cap X^{\perp }) =d$, corresponding stabilizer $S = S_X$ and associated ${\rm U}(V)$-module $\pi _X$, one has

\begin{align*} & {\rm Ext}^i_{{\rm U}(V)} \big( \pi_X, \omega_{V,\psi,\mu}\big)\\ &\quad \cong {\rm Ext}^i_{S/N(S)}\Big( \delta_{P/S}^{1/2} \cdot (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d},\, \delta_{S}^{1/2} \cdot |{\det}_{{\rm GL}_d}|^{-1/2}\mu \cdot \omega_{V_{n-2d},\psi,\mu} \Big) \end{align*}

where we note:

• • $S/ N(S) \cong {\rm GL}_d(E) \times {\rm U}_{a-d} \times {\rm U}_{b-d}$;

• • $(\pi _1)_{d,a-d}$ denotes the unnormalized Jacquet module of $\pi _1$ with respect to the $(d,a-d)$ parabolic subgroup in ${\rm GL}(V_a) \cong {\rm GL}_a(E)$, regarded as a representation of ${\rm GL}_d(E) \times {\rm U}_{a-d} \subset {\rm GL}_d(E) \times {\rm GL}_{a-d}(E)$ by restriction;

• • likewise, $(\pi _2)_{b-d,d}$ is the unnormalized Jacquet module of $\pi _2$ with respect to the $(b-d,d)$-parabolic subgroup in ${\rm GL}(V_b) \cong {\rm GL}_b(E)$, regarded as a representation of ${\rm U}_{b-d} \times {\rm GL}_d(E) \subset {\rm GL}_{b-d}(E) \times {\rm GL}_d(E)$ by restriction and taking contragredient on the ${\rm GL}_d(E)$ factor;

• • the characters $\delta _{P/S}$ and $\delta _S$ are trivial on ${\rm U}_{a-d} \times {\rm U}_{b-d}$ and are given on ${\rm GL}_d(E)$ by

  \begin{align*} \delta_{P/S} = |\!\det\!|^d \quad \text{and} \quad \delta_S= |\!\det\!|^{n-d}. \end{align*}

In particular, for the two open orbits $X$ corresponding to $d=0$, we have

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] \cong \sum_{\substack{i=j+k\\ V= V_a\oplus V_b}} {\rm Ext}^j_{{\rm U}(V_a)} [\pi_1|_{{\rm U}(V_a)}, \omega_{V_a, \psi,\mu}] \otimes {\rm Ext}^k_{{\rm U}(V_b)}[ \pi_2|_{{\rm U}(V_b)}, \omega_{V_b,\psi,\mu}] , \end{align*}

where $X = V_a$ are the isomorphism classes of non-degenerate subspaces of $V$ of dimension $a$ with orthogonal complement $X^{\perp } = V_b$.

Proof. For analyzing ${\rm Ext}^i_{{\rm U}(V)}[ \pi _X , \omega _{V,\psi, \mu }]$, we will need the following generalities on Ext groups and contragredients (cf. [Pra18] for generality (a) below).

1. (a) For any two smooth representations $U,V$ of a $p$-adic group $G$, we have

   \begin{align*} {\rm Ext}_G^i[ U,V^\vee] \cong {\rm Ext}_G^i[V, U^\vee] . \end{align*}

2. (b) For $H$ a closed subgroup of a $p$-adic group $G$, and $U$ any smooth representation of $H$ with smooth dual $U^\vee$,

   \begin{align*} [{\rm ind}_H^G U]^\vee \cong {\rm Ind}_H^G U^\vee. \end{align*}

3. (c) For a non-trivial character $\psi : F \rightarrow \mathbb {C}^\times$, with the associated Weil representation $\omega _{V,\psi, \mu }$ of ${\rm U}(V)$, we have

   \begin{align*} \omega_{V, \psi, \mu}^\vee \cong \omega_{V, \psi^-,\mu^{-1}}, \end{align*}
   where $\psi ^-(x) = \psi (-x)$.

We shall also use the following three lemmas whose proofs are left to the reader.

Lemma 4.5 Let $U \subset H$ be $p$-adic groups with $U$ a normal subgroup of $H$ which is a union of compact open subgroups. Let $\pi _1,\pi _2$ be two smooth representations of $H$ such that $U$ acts trivially on $\pi _2$. Then

\begin{align*} {\rm Ext}^i_H[\pi_1,\pi_2] \cong {\rm Ext}^i_{H/U}[ \pi_{1, U}, \pi_2]. \end{align*}

Lemma 4.6 Let $G$ be any $p$-adic group, $Z\subset G$ a closed central subgroup. If $\pi _1$ and $\pi _2$ are two smooth representations of $G$ on which $Z$ operates by different characters, then

\begin{align*} {\rm Ext}_G^i[\pi_1,\pi_2] = 0\quad \text{for all}\ i \geq 0. \end{align*}

Lemma 4.7 Let $V=V_n$ be a skew-Hermitian space over $E$ of dimension $n$, and let $\omega _{V,\psi,\mu }$ be a Weil representation of ${\rm U}(V)$. Let $Q_d = M_d N_d$ be the maximal parabolic subgroup of ${\rm U}(V)$ stabilizing a $d$-dimensional isotropic space (see § 4.5), so that $M_d \cong {\rm GL}_d(E) \times {\rm U}(V_{n-2d})$. Then for $Z_d$, the center of $N_d$, we have

\begin{align*} (\omega_{V,\psi,\mu})_{Z_d} = (\omega_{V,\psi, \mu})_{N_d} \cong (\mu \cdot |-|^{1/2} \circ \det) \otimes\omega_{{V_{n-2d}},\psi,\mu} , \end{align*}

as $M_d$-modules.

With these preliminaries in place, we now compute

\begin{alignat}{2} &\hskip -17pt{\rm Ext}^i_{\mathrm{U}(V)}[ \pi_X, \omega_{V,\psi, \mu}] & \notag \\ \quad\cong &\hskip 5pt {\rm Ext}^i_{\mathrm{U}(V)}[ {\rm ind}_{S}^{\mathrm{U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2}), \omega_{V,\psi, \mu}], &\text{ (by Proposition 4.3)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{\mathrm{U}(V)}[ \omega_{V,\psi^-, \mu^{-1}}, {\rm Ind}_{S}^{\mathrm{U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ], & \text{ (by (a), (b) and (c))} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S}[ \delta_{S}^{-1/2}\omega_{V,\psi^-, \mu^{-1}}, (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ], & \text{ (by Frobenius reciprocity)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/Z(S)}[ \delta_{S}^{-1/2}(\omega_{V,\psi^-, \mu^{-1}})_{Z(S)}, (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ], & \text{ (by Lemma 4.5)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/Z(S)}[ \delta_{S}^{-1/2} \cdot \mu^{-1} \cdot |\!\det\!|^{1/2} \cdot \omega_{{V_{n-2d}},\psi^-, \mu^{-1}}, (\pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2})^\vee ] & \text{ (by Lemma 4.7)} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/Z(S)}[ \pi_1 \otimes \pi_2 \otimes \delta_{P/S}^{1/2}, \,\, \delta_{S}^{1/2} |\!\det\!|^{-1/2}\cdot \mu \cdot \omega_{V_{n-2d},\psi,\mu}] & \hskip 5pt \text{(by (a))} \notag \\ \quad\cong &\hskip 5pt{\rm Ext}^i_{S/N(S)}[ (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} \otimes \delta_{P/S}^{1/2},\,\, \delta_{S}^{1/2} |\!\det\!|^{-1/2} \cdot \mu \cdot \omega_{V_{n-2d},\psi,\mu}] & \notag \end{alignat}

where $(\pi _1)_{d,a-d}$ denotes the unnormalized Jacquet module of $\pi _1$ with respect to the $(d,a-d)$ parabolic subgroup in ${\rm GL}_a(E)$; similarly for $(\pi _2)_{b-d,d}$. Here we have applied Lemma 4.5 (taking $U = N(S)/Z(S)$) for the last isomorphism for which it is important to note that $N(S)/Z(S)$ maps isomorphically to the product of the unipotent radicals of the $(d,a-d)$-parabolic subgroup of ${\rm GL}_a(E)$ and the $(b-d,d)$-parabolic subgroup of ${\rm GL}_b(E)$.

For the final assertion in the proposition regarding the open orbits corresponding to $d=0$, it suffices to observe that for the direct sum of non-degenerate skew-Hermitian spaces $V = V_a \oplus V_b$, we have the tensor product decomposition of their Weil representations:

\begin{align*} \omega_{V,\psi,\mu} \cong \omega_{V_a,\psi,\mu} \otimes \omega_{V_b,\psi,\mu}, \end{align*}

as representations of ${\rm U}(V_a) \times {\rm U}(V_b) \subset {\rm U}(V)$. Thus, the final assertion is a direct consequence of the Kunneth theorem [Pra18, Theorem 3.1], completing the proof of Proposition 4.4.

4.8 Branching for $\pi _1 \times \pi _2$

We can now assemble the results of Proposition 4.4 for the various ${\rm U}(V)$-orbits $[X]$ on ${\rm GL}(V)/ P$ to understand the branching problem

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \pi_1 \times \pi_2, \omega_{V, \psi, \mu}). \end{align*}

The result is most definitive when only the open orbits have non-zero contribution. The following theorem, which is the main result of this section, gives a simple sufficient condition (temperedness of $\pi _1$ and $\pi _2$) for this to happen.

Theorem 4.8 Suppose that $\pi _1$ and $\pi _2$ are tempered representations of ${\rm GL}(V_a)$ and ${\rm GL}(V_b)$ (with unitary central characters), so that $\pi = \pi _1 \times \pi _2$ is a tempered principal series representation of ${\rm GL}(V)$ for $V=V_a+ V_b$. If $[X]$ is a non-open orbit of ${\rm U}(V)$ on ${\rm GL}(V)/P$, then for $\pi _X$, the subquotient of $\pi$ supported on the orbit $[X]$, we have

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] = 0, \end{align*}

for all $i \geq 0$.

As a consequence, for all $i \geq 0$, one has

(4.10)\begin{align} &\bigoplus_{\delta} {\rm Ext}^i_{{\rm U}(V_{\delta})}[ \pi, \omega_{V_{\delta},\psi, \mu}] \nonumber\\ &\quad \cong \bigoplus_{i=j+k} \bigg(\bigoplus_{\delta'} {\rm Ext}^j_{{\rm U}(V_{a,\delta'})} [ \pi_1, \omega_{V_{a, \delta'},\psi, \mu}] \bigg) \otimes \bigg(\bigoplus_{\delta''} {\rm Ext}^k_{{\rm U}(V_{b,\delta''})}[ \pi_2, \omega_{V_{b,\delta''},\psi, \mu}] \bigg), \end{align}

where the sums over $\delta$, $\delta '$, and $\delta ''$ run over $F^{\times }/NE^{\times }$ or $E_0^{\times }/ NE^{\times }$ according to the parity of $n, a,b$, respectively.

Hence, for $i=0$, one has

(4.11)\begin{align} &{\rm Hom}_{{\rm U}(V_{\delta})}[ \pi, \omega_{V_{\delta},\psi}]\nonumber\\ &\quad \cong \bigoplus_{(\delta', \delta''): V_{a,\delta'} \oplus V_{b,\delta''} \cong V_{\delta} } {\rm Hom}_{{\rm U}(V_{a,\delta'})}[ \pi_1, \omega_{V_{a,\delta'},\psi}] \otimes {\rm Hom}_{{\rm U}(V_{b,\delta''})} [ \pi_2, \omega_{V_{b,\delta''},\psi}] . \end{align}

In particular,

(4.12)\begin{align} &\bigoplus_{\delta} {\rm Hom}_{{\rm U}(V_{\delta})}[ \pi, \omega_{V_{\delta},\psi}]\nonumber\\ &\quad \cong \bigg(\bigoplus_{\delta'} {\rm Hom}_{{\rm U}(V_{a,\delta'})}[ \pi_1, \omega_{V_{a,\delta'},\psi}] \bigg)\otimes \bigg(\bigoplus_{\delta''} {\rm Hom}_{{\rm U}(V_{b,\delta''})} [ \pi_2, \omega_{V_{b,\delta''},\psi}] \bigg). \end{align}

Proof. By Proposition 4.4, ${\rm Ext}^i_{{\rm U}(V)}[ \pi _X, \omega _{V,\psi,\mu }]$ is equal to

\begin{align*} {\rm Ext}^i_{S/N(S)}[ \delta_{P/S}^{1/2} \cdot (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} , \delta_{S}^{1/2} \cdot |\!\det\!|^{-1/2} \mu\cdot \omega_{V_{n-2d},\psi,\mu}] . \end{align*}

Since $\pi _1$ is tempered, it follows by Casselman's temperedness criterion that the central exponents of $(\pi_1)_{d, a-d}$ have the form $\delta_{P_{d, a-d}}^{(1 + \alpha)/2}$ with $\alpha \geq 0$. Moreover, for $(g,h) \in {\rm GL}_d(E) \times {\rm U}_{a-d}(F)$,

\begin{align*} \delta_{P_{d,a-d}}(g,h)^{(1+\alpha)/2} = |\!\det(g)|^{(a-d+\epsilon)/2 }, \quad \text{with $\epsilon=\alpha \cdot (a-d)$.} \end{align*}

Similarly, the central exponents of $(\pi_2)_{b-d,d}$ have the form

\begin{align*} \delta_{P_{b-d,d}}^{(1 + \alpha')/2} \quad \text{with}\ \alpha' \geq 0,\end{align*}

and for $(h, (g^*) ^{-1} ) \in {\rm U}_{b-d}(F) \times {\rm GL}_d(E)$,

\begin{align*} \delta_{P_{b-d,d}}(h, (g^*)^{-1})^{(1+\alpha')/2} = |\!\det(g)|^{(b-d+\epsilon')/2}, \quad \text{with $\epsilon' = \alpha' \cdot (b-d)$.} \end{align*}

Summarizing, we have:

1. (1) the representation

   \begin{align*} A = \delta_{P/S}^{1/2} \cdot (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} \end{align*}
   has central exponents of the form
   \begin{align*} |\!\det \!|^{(a-d+\epsilon)/2} \cdot |\!\det \!|^{(b-d+\epsilon')/2} |\!\det \!|^{d/2} = |\!\det \!|^{(n-d+\epsilon+\epsilon')/2} \end{align*}
   with $\epsilon$ and $\epsilon'$ non-negative;

2. (2) the representation

   \begin{align*} B= \delta_{S}^{1/2} \cdot |\!\det \!|^{-1/2} \mu\cdot \omega_{V_{n-2d},\psi,\mu} \end{align*}
   is the twist of a unitary representation of ${\rm GL}_d(E) \times {\rm U}(V_{n-2d})$ by the character
   \begin{align*} |\!\det\!|^{(n-d)/2} \cdot |\!\det\! |^{-1/2} = |\!\det\!|^{(n-d-1)/2} \end{align*}
   of ${\rm GL}_d(E)$.

Thus, when $d > 0$, the actions of the center of ${\rm GL}_d(E)$ in

\begin{align*} S/N(S)={\rm GL}_d(E) \times {\rm U}(V_{a-d}) \times {\rm U}(V_{b-d}) \end{align*}

on the two representations $A$ and $B$ are different. Therefore, by Lemma 4.6,

\begin{align*} {\rm Ext}^i_{S/N(S)}[A,B] = 0, \end{align*}

for all $i \geq 0$ (as long as $d\not = 0$). This completes the proof that for a non-open ${\rm U}(V)$-orbit $[X] \subset {\rm GL}(V)/P$, the associated subquotient $\pi _X$ of the ${\rm U}(V)$-module $\pi$ satisfies

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi}] = 0 \quad \text{for all $i \geq 0$.} \end{align*}

As a consequence of the vanishing of ${\rm Ext}^i$, $i \geq 0$, for all non-open orbits, we deduce that

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] = \bigoplus_X {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}], \end{align*}

where now $X$ runs over the two open orbits of ${\rm U}(V)$ on ${\rm GL}(V)/P$. Since Proposition 4.4 calculates ${\rm Ext}^i_{{\rm U}(V)}[ \pi _X, \omega _{V,\psi }]$ for the open orbits, the proof of Theorem 4.8 is complete.

5. Application to Conjecture 2.1

In this section, we deduce the implications of Theorem 4.8 for Conjecture 2.1. Indeed, we shall show how Theorem 4.8 allows us to reduce Conjecture 2.1 for tempered representations to the case of discrete-series representations of ${\rm GL}_n(E)$. This allows us to prove Conjecture 2.1 for unitary principal series induced from the Borel subgroup. We also investigate whether the Mackey theory argument allows one to further reduce Conjecture 2.1 to the case of supercuspidal representations. As we shall see, we fall slightly short of that, but we will at least be able to prove Conjecture 2.1 for the Steinberg representation of ${\rm GL}_n(E)$.

5.1 Inductive argument

Let us first record the following consequence of Theorem 4.8.

Corollary 5.1 Let $V = V_a \oplus V_b$ be a direct sum of non-degenerate skew-Hermitian spaces, and let $\pi _1$ and $\pi _2$ be irreducible tempered representations of ${\rm GL}(V_a)$ and ${\rm GL}(V_b)$, respectively. Then we have the following.

1. (i) If Conjecture 2.1 holds for $\pi _1$ and $\pi _2$, then Conjecture 2.1 holds for the unitary principal series representation $\pi _1 \times \pi _2$ of ${\rm GL}(V)$.

2. (ii) If

   \begin{align*} {\rm Ext}^i_{{\rm U}(V_a)}(\pi_1, \omega_{V_a, \psi,\mu}) = {\rm Ext}^i_{{\rm U}(V_b)}(\pi_2, \omega_{V_b,\psi,\mu}) = 0 \quad \text{ for all $i>0$,} \end{align*}
   then
   \begin{align*} {\rm Ext}^i_{{\rm U}(V)}(\pi_1 \times \pi_2 , \omega_{V,\psi, \mu}) = 0 \quad \text{ for all $i>0$.} \end{align*}

Proof. The vanishing statement in part (ii) follows from Theorem 4.8, especially (4.9). Likewise, (4.11) imply that Conjecture 2.1(i) and (ii) for $\pi _1 \times \pi _2$ follows from the corresponding statements for $\pi _1$ and $\pi _2$. Thus, it remains to verify that the unique skew-Hermitian space $V_{\delta }$ for which ${\rm Hom}_{{\rm U}(V_{\delta })}(\pi _1 \times \pi _2 , \omega _{V_{\delta }, \psi, \mu })$ is non-zero is as predicted by Conjecture 2.1(iii).

Assume then that

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\pi_1 \times \pi_2 , \omega_{V, \psi, \mu}) \ne 0. \end{align*}

By (4.10),

\begin{align*} {\rm Hom}_{{\rm U}(V_a)}(\pi_1 , \omega_{V_a, \psi, \mu}) \ne 0 \quad \text{and} \quad {\rm Hom}_{{\rm U}(V_b)}(\pi_2 , \omega_{V_b, \psi, \mu}) \ne 0 \end{align*}

for a unique pair of skew-Hermitian spaces $V_a$ and $V_b$ satisfying $V_a \oplus V_b \cong V$. As we have assumed that Conjecture 2.1(iii) holds for $\pi _1$ and $\pi _2$, we have

\begin{align*} \mu(\det(V_a)) = \epsilon( 1/2, \pi_1 \times {}^\sigma\pi_1 ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi_1}(-1)^a \cdot \omega_{E/F}(-1)^{a(a-1)/2}, \end{align*}

and

\begin{align*} \mu(\det(V_b)) = \epsilon( 1/2, \pi_2 \times {}^\sigma\pi_2 ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi_2}(-1)^b \cdot \omega_{E/F}(-1)^{b(b-1)/2}. \end{align*}

This implies that for $\pi = \pi _1 \times \pi _2$,

\begin{align*} \mu(\det(V)) = \epsilon( 1/2, \pi \times {}^\sigma\pi ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi}(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

using the facts that $\mu ={}^\sigma \mu ^{-1}$, $\mu (-1)= \omega _{E/F}(-1)$ and

\begin{align*} \epsilon (\Pi + {}^\sigma\Pi ^\vee, \psi_E) = \det \Pi(-1) \end{align*}

for any representation $\Pi$ of ${\rm GL}_m(E)$. This completes the proof of the corollary.

5.2 Reduction to the discrete-series case

Corollary 5.1 allows one to reduce Conjecture 2.1 for tempered representations to the case of discrete-series representations.

Corollary 5.2 If Conjecture 2.1 holds for all (unitary) discrete-series representations of all ${\rm GL}(V)$, then it holds for all (unitary) tempered representations of all ${\rm GL}(V)$.

Proof. This follows from Corollary 5.1 and the fact that any tempered non-discrete-series representation of ${\rm GL}(V)$ is irreducibly and unitarily induced from a discrete-series representation of a proper parabolic subgroup.

5.3 Borel–Principal series

In addition, by applying Corollary 5.1 inductively, we deduce the following.

Corollary 5.3 Conjecture 2.1 holds for all irreducible unitary principal series representations of ${\rm GL}(V)$ induced from a unitary character of a Borel subgroup.

Proof. Using Corollary 5.1, this follows by induction on $\dim V$. The base case, with $\dim V = 1$, is Theorem 3.1 due to Moen and Rogawski.

5.4 An alternative proof

We now give another proof of Corollary 5.3 as it brings out an interesting structure of the open orbits of ${\rm U}(V)$ on ${\rm GL}(V)/B$.

Suppose that

\begin{align*} \Pi = {\rm Ind}_B^{{\rm GL}(V)} (\chi_1 \otimes \cdots \otimes \chi_n) \quad \text{(normalized induction),} \end{align*}

so that its $L$-parameter is

\begin{align*} M = \bigoplus_i \chi_i . \end{align*}

On restriction to ${\rm U}(V)$, Theorem 4.8 inductively implies that only the open ${\rm U}(V)$-orbits on the flag variety ${\rm GL}(V)/B$ will contribute to the Hom space ${\rm Hom}_{{\rm U}(V)}[ \Pi, \omega _{V,\psi }]$. Moreover, using Lemma 4.2 inductively, the open orbits can be described as follows. Given an ordered collection

\begin{align*} \mathcal{L} = \{L_1,\ldots , L_n \} \end{align*}

of non-degenerate orthogonal lines in $V$, the ${\rm U}(V)$-orbit of the flag

\begin{align*} \mathfrak{F}_{\mathcal{L}}: L_1 \subset L_1 \oplus L_2 \subset \cdots \end{align*}

is an open orbit, and the stabilizer of $\mathfrak {F}_{\mathcal {L}}$ in ${\rm U}(V)$ is the subgroup

\begin{align*} {\rm U}(\mathcal{L}):= \prod_i {\rm U}(L_i). \end{align*}

Moreover, all open orbits are given by such an ordered collection $\{ L_i \}$ of isomorphism classes of (non-degenerate) skew-Hermitian $E$-spaces of dimension 1, subject to the condition that $\bigoplus _i L_i \cong V$ as skew-Hermitian $E$-spaces; we say that such an $\mathcal {L}$ is $V$-relevant. There are thus $2^{n-1}$ open orbits, indexed by $V$-relevant $\mathcal {L}$. This can also be gleaned from a Galois cohomological argument: having fixed an open orbit over $F$ with stabilizer ${\rm U}(\mathcal {L})$ in ${\rm U}(V)$ and noting that there is exactly one open orbit over $\overline {F}$, the number of open ${\rm U}(V)$-orbits is given by

\begin{align*} {\rm Ker}\big( H^1(F, {\rm U}(\mathcal{L})) \rightarrow H^1(F, {\rm U}(V))\big) = {\rm Ker} \big( (F^{\times}/ N(E^{\times}))^n \rightarrow F^{\times}/ N(E^{\times}) \big). \end{align*}

Hence, by an inductive application of Theorem 4.8, we have

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V,\psi,\mu}) \cong \bigoplus_{\mathcal{L}} {\rm Hom}_{{\rm U}(V)} ({\rm ind}_{{\rm U}(\mathcal{L})}^{{\rm U}(V)} ( \boxtimes_i \chi_i ), \omega_{V,\psi,\mu}), \end{align*}

where the sum runs over $V$-relevant $\mathcal {L}$. By Frobenius reciprocity, and the fact that

\begin{align*} \omega_{V,\psi, \mu} |_{{\rm U}(\mathcal{L})} \cong \bigotimes_i \omega_{L_i,\psi, \mu}, \end{align*}

one deduces that

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V,\psi,\mu}) \cong \bigoplus_{\mathcal{L}} \bigotimes_i {\rm Hom}_{{\rm U}(L_i)}( \chi_i, \omega_{L_i,\psi, \mu}). \end{align*}

Now by Theorem 3.1 (the theorem of Moen and Rogawski),

\begin{align*} {\rm Hom}_{{\rm U}(L_i)}( \chi_i, \omega_{L_i,\psi, \mu}) \ne 0 \Longleftrightarrow \epsilon(1/2, \chi_i/ \chi_i^{\sigma} \cdot \mu^{-1}, \psi_E) \cdot \chi_i(-1) = \mu(\det(L_i)). \end{align*}

Hence, at most one term in the sum over $\mathcal {L}$ has non-zero contribution, and this unique $\mathcal {L}$ exists if and only if

\begin{align*} \mu(\det(V)) = \prod_i \epsilon(1/2, \chi_i/ \chi_i^{\sigma} \cdot \mu^{-1}, \psi_E) \cdot \chi_i(-1). \end{align*}

To prove Conjecture 2.1, we explicate as follows:

\begin{align*} &\epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \cdot \mu^{-1}, \psi_E) \\ &\quad = \prod_i \epsilon(1/2, \chi_i/\chi_i^{\sigma} \cdot \mu^{-1},\psi_E) \cdot \prod_{i< j} \epsilon\big( 1/2, (\chi_i/ \chi_j^{\sigma} + \chi_j/ \chi_i^{\sigma}) \cdot \mu^{-1}, \psi_E \big). \end{align*}

For $i < j$, observe that

\begin{align*} \epsilon\big( 1/2, (\chi_i/ \chi_j^{\sigma} + \chi_j/ \chi_i^{\sigma}) \cdot \mu^{-1}, \psi_E \big) &= \epsilon ( 1/2, \chi_i/ \chi_j^{\sigma}\cdot \mu^{-1}, \psi_E) \epsilon ( 1/2, \chi^\sigma_j/ \chi_i\cdot \mu^{-1}, \psi_E) \\ & = \chi_i(-1) \cdot \chi_j(-1) \cdot \omega_{E/F}(-1), \end{align*}

where we have used the following standard properties of the epsilon factor:

1. (1) $\epsilon (1/2, W, \psi _E) \cdot \epsilon (1/2, W^\vee, \psi _E) = \det (W)(-1)$;

2. (2) $\epsilon (1/2, W, \psi _E) = \epsilon (1/2, W^\sigma, \psi _E)$.

It follows that

\begin{align*} \prod_{i< j} \epsilon\big( 1/2, (\chi_i/ \chi_j^{\sigma} + \chi_j/ \chi_i^{\sigma}) \cdot \mu^{-1}, \psi_E \big) = \det(M)(-1)^{n-1} \cdot \omega_{E/F}(-1)^{n(n-1)/2}. \end{align*}

Putting everything together, we see that ${\rm Hom}_{{\rm U}(V)}(\Pi, \omega _{V,\psi,\mu }) \ne 0$ if and only if

\begin{align*} \mu(\det(V)) = \epsilon(1/2, M \otimes {}^\sigma \!M ^\vee \cdot \mu^{-1}, \psi_E) \cdot \det(M)(-1)^n \cdot \omega_{E/F}(-1)^{n(n-1)/2}, \end{align*}

as desired.

5.5 Reduction to the supercuspidal case

We have seen that Conjecture 2.1 for tempered representations can be reduced to the case of discrete-series representations by a Mackey theory argument. In the rest of the section, we investigate whether the same argument can be used to reduce Conjecture 2.1 for discrete-series representations to the case of supercuspidal representations. It turns out that this can be done under a certain hypothesis. While we cannot prove this hypothesis in general, it can be shown in some situations. This will allow us to prove Conjecture 2.1 for the Steinberg representation, for example.

Let us first set up some notation and formulate the relevant hypothesis. Suppose that $\pi$ is a supercuspidal representation (with unitary central character) of ${\rm GL}(V) = {\rm GL}_m(E)$. The parabolically induced representation

\begin{align*} \pi |\!\det\!|^{({n-1})/{2}} \times \pi|\!\det\!|^{({n-3})/{2}} \times \cdots\times \pi|\!\det\!|^{-({n-1})/{2}} \end{align*}

of ${\rm GL}(V^{\oplus n}) \cong {\rm GL}_{mn}(E)$ is a standard module and, thus, has a unique irreducible quotient ${\rm Sp}(\pi,n)$, which is often called a Speh representation and is non-tempered (if $n>1$). This parabolically induced representation also has a unique irreducible submodule ${\rm St}(\pi,n)$; this is the ‘generalized Steinberg’ representation, which is a discrete-series representation. All the irreducible (unitary) discrete-series representations of general linear groups are of the form ${\rm St}(\pi, n)$. The supercuspidal ones are precisely those with $n=1$.

If

\begin{align*} \phi_\pi: W_E\rightarrow {\rm GL}_m(\mathbb{C}) \end{align*}

is the $L$-parameter of $\pi$, then the $L$-parameter of ${\rm St}(\pi,n)$ is the representation $\phi _{\pi } \otimes {\rm Sym}^{n-1}(\mathbb {C}^2)$ of the Weil–Deligne group $WD_E= W_E \times {\rm SL}_2(\mathbb {C})$. We also write $[n]={\rm Sym}^{n-1}(\mathbb {C}^2)$ for the unique irreducible $n$-dimensional representation of ${\rm SL}_2(\mathbb {C})$, and write $\phi _{\pi }[n]$ for the $L$-parameter of ${\rm St}(\pi,n)$.

To deal with the generalised Steinberg representations, we will need to make an assumption. In this, $V,W$ are the two isomorphism classes of skew-Hermitian spaces over $E$ of dimension $m$. Then we make the following assumption:

\begin{align*} ({\bf Assumption}) \quad \begin{cases} {\rm Hom}_{{\rm U}(V+ V )}[{\rm Sp}(\pi,2), \omega_{V+V,\psi, \mu}] = 0, \\ {\rm Hom}_{{\rm U}(V+ W )}[{\rm Sp}(\pi,2), \omega_{V+W,\psi, \mu}] = 0. \end{cases} \end{align*}

We remark that this assumption is a case of the non-tempered twisted GGP conjecture formulated in Conjecture 7.2.

With this assumption formulated, our result is as follows.

Theorem 5.4 Let $\pi$ be a supercuspidal representation of ${\rm GL}(V)$ with $\dim V = m$. If $\pi$ satisfies Conjecture 2.1 and the above (Assumption), then Conjecture 2.1 holds for the discrete-series representations ${\rm St}(\pi, n)$ for all $n \geq 1$.

We make this more precise as follows.

1. (a) Suppose that $V$ and $W$ are the two isomorphism classes of skew-Hermitian spaces over $E$ of dimension $m$ and

   \begin{align*} {\rm Hom}_{{\rm U}(V)}(\pi, \omega_{V,\psi, \mu}) & \cong \mathbb{C}, \\ {\rm Hom}_{{\rm U}(W)}(\pi, \omega_{W,\psi, \mu}) & = 0. \end{align*}
   Then, under (Assumption), one has
   \begin{align*} {\rm Hom}_{{\rm U}(V^n)}({\rm St}(\pi,n), \omega_{V^n,\psi, \mu}) & \cong \mathbb{C}, \\ {\rm Hom}_{{\rm U}(W+V^{n-1})}({\rm St}(\pi,n), \omega_{W+V^{n-1},\psi, \mu}) & = 0, \end{align*}
   for the two isomorphism classes of skew-Hermitian spaces $V^n, W+V^{n-1}$ of dimension $mn$ over $E$,

2. (b) If

   \begin{align*} \mu(\det(V)) \stackrel{(1)}= \epsilon( 1/2, \phi_{\pi} \times {}^\sigma\phi_{\pi} ^\vee \times \mu^{-1}, \psi_E) \cdot \omega_{\pi}(-1)^m \cdot \omega_{E/F}(-1)^{m(m-1)/2}, \end{align*}
   then for the skew-Hermitian space $V^{ n}$,
   \begin{align*} \mu(\det(V^{ n})) & = \mu(\det(V)^n) \\ & \stackrel{(2)}= \epsilon( 1/2, \phi_{\pi}[n] \times {}^\sigma\phi_{\pi} ^\vee[n] \times \mu^{-1}, \psi_E) \cdot \omega_{\pi}(-1)^{nm} \cdot \omega_{E/F}(-1)^{mn(mn-1)/2}. \end{align*}

Proof. The first assertion of the theorem (concerning the truth of Conjecture 2.1) is an immediate consequence of statements (a) and (b). We shall prove these two statements in turn, starting with the simpler statement (b).

Proof of Theorem 5.4(b) Recall from [Tat79] that for an irreducible representation $\lambda \otimes [n]$ of $WD_E=W_E \times {\rm SL}_2(\mathbb {C})$, one has

\begin{align*} \epsilon( \lambda \otimes [n]) = \epsilon(\lambda)^n \cdot \det (-F, \lambda^I)^{n-1}, \end{align*}

where $\lambda ^I$ denotes the subspace of $\lambda$ fixed by the inertia group $I$ and $F$ denotes the Frobenius element of $W_E/I$.

On the other hand, by the Clebsch–Gordon theorem,

\begin{align*} [n] \otimes [n] = [2n-1] \oplus [2n-3] \oplus \cdots \oplus [1]. \end{align*}

In particular, only odd integers $(2d+1)$ appear in this decomposition. It is easy to see that in the expression

\begin{align*} \epsilon( \lambda \otimes [2d+1]) = \epsilon(\lambda)^{2d+1} \cdot \det (-F, \lambda^I)^{2d}, \end{align*}

the factor $\det (-F, \lambda ^I)^{2d}$ is trivial for $\lambda$ a conjugate selfdual representation of $W_E$. Hence, we find that

\begin{align*} \epsilon( \lambda \otimes [2d+1]) = \epsilon(\lambda)^{2d+1} \end{align*}

for $\lambda$ a conjugate selfdual representation of $W_E$. These considerations, applied to the conjugate selfdual representation $\lambda$ of $W_E$ associated to $\pi \times {}^\sigma \pi ^\vee \times \mu ^{-1}$, allow one to prove identity $(2)$ from identity $(1)$; we leave the simple and pleasant computation to the reader.

Proof of Theorem 5.4(a) The proof of statement (a) depends on some intermediate results contained in the following series of lemmas.

Lemma 5.5 Let $\pi$ be a unitary supercuspidal representation of ${\rm GL}(V) \cong {\rm GL}_m(E)$.

1. (i) One has a short exact sequence ${\rm GL}_{mn}(E)$-representations:

   \begin{align*} 0 \rightarrow K_n \rightarrow \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) \rightarrow {\rm St}(\pi, n) \rightarrow 0, \end{align*}
   with $K_n$ an irreducible representation of ${\rm GL}_{mn}(E)$ and we have written $\nu$ for the character $|\!\det\!|$.

2. (ii) The irreducible representation $K_n$ of ${\rm GL}_{mn}(E)$ arising in the exact sequence above, sits in the following short exact sequence:

   \begin{align*} 0 \rightarrow L_{n-1} \rightarrow \nu^{-(n-2)/2}{\rm Sp}(\pi,2) \times \nu {\rm St}(\pi,n-2) \rightarrow K_n \rightarrow 0. \end{align*}

Proof. (i) The fact that the discrete-series representation ${\rm St}(\pi, n)$ of ${\rm GL}_{mn}(E)$ appears as a quotient of the principal series $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$ is clear, since ${\rm St}(\pi, n)$ is a quotient of the principal series representation

\begin{align*} \nu^{-(n-1)/2}\pi \times \nu^{-(n-3)/2} \pi \times \cdots \times \nu^{(n-1)/2}\pi . \end{align*}

It is well-known from Zelevinski [Zel80] that the principal series $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$ has length 2, so that $K_n$ is irreducible and part (i) is proved.

(ii) Since $K_n$ is irreducible, it suffices to prove that there is a non-zero ${\rm GL}_{mn}(E)$-equivariant homomorphism from the principal series $\nu ^{-(n-2)/2}{\rm Sp}(\pi,2) \times \nu {\rm St}(\pi,n-2)$ to $K_n$. By the second adjointness theorem, this boils down to proving that the normalized Jacquet functor of $K_n$ for the opposite parabolic of the maximal standard parabolic with Levi ${\rm GL}_{2m}(E) \times {\rm GL}_{(n-2)m}(E)$ contains the irreducible representation $\nu ^{-(n-2)/2}{\rm Sp}(\pi,2) \otimes \nu {\rm St}(\pi,n-2)$ of ${\rm GL}_{2n}(E) \times {\rm GL}_{(m-2)n}(E)$ as a submodule. We leave this simple calculation to the reader.

Next, we apply Proposition 4.4 to the two principal series representations appearing in Lemma 5.5. We do not perform the explicit calculation here, but simply summarize the results.

Lemma 5.6 Let $P$ denote the maximal parabolic subgroup of ${\rm GL}(V^n)$ from which the principal series representation considered below is induced. Then we have the following.

1. (i) For any non-open ${\rm U}(V^{n})$-orbit $[X] \subset {\rm GL}(V^{ n})/P$, the associated subquotient $\pi _X$ of the ${\rm U}(V^{n})$-module $\nu ^{-(n-2)/2}{\rm Sp}(\pi,2) \times \nu {\rm St}(\pi,n-2)$, satisfies

   \begin{align*} {\rm Ext}^i_{{\rm U}(V^{ n})}[ \pi_X, \omega_{V^{ n},\psi}] = 0, \end{align*}
   for all $i \geq 0$. If (Assumption) holds for $\pi$, then the above result holds for the open orbits as well and therefore
   \begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}[\nu^{-(n-2)/2}{\rm Sp}(\pi,2) \times \nu {\rm St}(\pi,n-2), \omega_{V^{ n},\psi}] = 0. \end{align*}

2. (ii) For any non-open ${\rm U}(V^{n})$-orbit $[X] \subset {\rm GL}(V^{ n})/P$, the associated subquotient $\pi _X$ of the ${\rm U}(V^{n})$-module $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$, satisfies

   \begin{align*} {\rm Ext}^i_{{\rm U}(V^n)}[ \pi_X, \omega_{V^{ n},\psi}] = 0, \end{align*}
   for all $i \geq 0$.

3. (iii) For any non-open ${\rm U}(V^{n})$-orbit $[X] \subset {\rm GL}(V^{ n})/P$, the associated subquotient $\pi _X$ of the ${\rm U}(V^{n})$-module $\nu ^{1/2} {\rm St}(\pi, n-1) \times \nu ^{-(n-1)/2}\pi$, satisfies

   \begin{align*} {\rm Ext}^i_{{\rm U}(V^n)}[ \pi_X, \omega_{V^{ n},\psi}] = 0, \end{align*}
   for all $i \geq 0$.

With the above two lemmas at hand, let us now return to the proof of Theorem 5.4(a). By Lemma 5.5(ii), combined with Lemma 5.6(i), we deduce that

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}[K_n,\omega_{V^{ n},\psi}] = 0. \end{align*}

Therefore, by Lemma 5.5(i),

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}({\rm St}(\pi, n) , \omega_{V^{ n},\psi}) \cong {\rm Hom}_{{\rm U}(V^{ n})}(\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V^{ n},\psi}). \end{align*}

Furthermore, from Lemma 5.6(ii),

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}(\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V^{ n},\psi}) \end{align*}

is contributed by the submodule of the principal series representation

\begin{align*} \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) \end{align*}

supported on the open orbits.

Observe that the open orbits of the action of ${\rm U}(V^n)$ on ${\rm GL}_{mn}(E)/P_{m,m(n-1)}$ are parametrized by the isomorphism classes of the skew-Hermitian subspaces of $V^n$ of dimension $m=\dim (V)$. Thus, there are exactly two orbits, represented by the skew-Hermitian spaces $V$ and $W$, with stabilizer in ${\rm U}(V^n)$ being ${\rm U}(V) \times {\rm U}(V^{n-1})$ and ${\rm U}(W) \times {\rm U}(W+V^{n-2})$. Therefore,

\begin{align*} {\rm Hom}_{{\rm U}(V^{ n})}({\rm St}(\pi, n) , \omega_{V^{ n},\psi}) \cong {\rm Hom}_{{\rm U}(V^{ n})}(\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V^{ n},\psi}) \end{align*}

is the sum $A + B$ of the contributions coming from these two open orbits, with

\begin{align*} A & = {\rm Hom}_{{\rm U}(V)}(\pi, \omega_{V,\psi}) \otimes {\rm Hom}_{{\rm U}(V^{n-1})}({\rm St}(\pi, n-1) , \omega_{V^{(n-1)},\psi}), \\ B & = {\rm Hom}_{{\rm U}(W)}(\pi, \omega_{W,\psi}) \otimes {\rm Hom}_{{\rm U}(W+V^{n-2})}({\rm St}(\pi, n-1) , \omega_{W+V^{(n-2)},\psi}). \end{align*}

Now as ${\rm Hom}_{{\rm U}(W)}(\pi, \omega _{W,\psi })=0$, we conclude that $B = 0$. This completes the proof of part (a) of Theorem 5.4, and hence the proof of Theorem 5.4 is complete.

5.6 The Steinberg representation

At the moment, we do not know how to prove the (Assumption) that

\begin{align*} {\rm Hom}_{{\rm U}(V^{ 2})}[{\rm Sp}(\pi,2), \omega_{V^{ 2},\psi}] = {\rm Hom}_{{\rm U}(V+W)}[{\rm Sp}(\pi,2), \omega_{V+W,\psi}] = 0, \end{align*}

except when $m=\dim V = \dim W =1$, i.e. when $\pi$ is a character of $E^\times$. In this case, the (Assumption) is equivalent to saying that the Weil representation of ${\rm U}(2)$ does not contain any one-dimensional representation of ${\rm U}(2)$ where ${\rm U}(2)$ is either of the two unitary groups in two variables. The next lemma establishes this.

Lemma 5.7 For $V$ be a skew-Hermitian space over $E$, a non-archimedean local field, of dimension $d \geq 2$, one has

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \chi, \omega_{V,\psi, \mu}] = 0 \end{align*}

for any one-dimensional character $\chi$ of ${\rm U}(V)$.

Proof. If $V = V_1\oplus V_2$, a direct sum of skew-Hermitian spaces, one knows that as representations of ${\rm U}(V_1) \times {\rm U}(V_2) \subset {\rm U}(V)$,

\begin{align*} \omega_{V,\psi, \mu}= \omega_{V_1,\psi,\mu} \otimes \omega_{V_2,\psi,\mu}. \end{align*}

Therefore, the proof of the lemma reduces to the case of $d=2$.

When $d =2$, we have seen in the discussion in § 3.2 that $\omega _{V, \psi,\mu }$ is a direct sum of irreducible summands (with different central characters), each of which has dihedral $L$-parameters. Hence, $\omega _{V, \psi,\mu }$ does not contain one-dimensional characters of ${\rm U}(V)$. (This uses the requirement for $E$ to be a non-archimedean local field.)

As a consequence, we obtain the following.

Corollary 5.8 The Steinberg representation ${\rm St}$ of ${\rm GL}(V)$ satisfies Conjecture 2.1.

5.7 Ext vanishing

The results of this section also prove the following theorem on the vanishing of Ext groups for tempered representations.

Theorem 5.9 Let $F$ be a non-archimedean local field and $E$ a separable quadratic algebra over $F$. Let $V$ be a skew-Hermitian space over $E$, with corresponding unitary group ${\rm U}(V) \subset {\rm GL}(V)$. For any irreducible tempered representation $\Pi$ of ${\rm GL}(V)$ and any Weil representation $\omega _{V, \psi, \mu }$ of ${\rm U}(V)$,

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}( \Pi, \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1. \end{align*}

Proof. ¹ As any irreducible tempered representation of ${\rm GL}(V)$ is parabolically induced from an irreducible discrete-series representation of a Levi subgroup, by Corollary 5.1, it suffices to prove this theorem for the discrete-series representations $\Pi = {\rm St}(\pi,n)$ of ${\rm GL}_{mn}(E)$ where $\pi$ is a cuspidal representation of ${\rm GL}_m(E)$. We prove this by an induction on the integer $n$.

The base case $n=1$ is clear since a supercuspidal representation $\pi$ of ${\rm GL}_m(E)$ is a projective representation when restricted to any subgroup $H \subset {\rm GL}_m(E)$ for which the intersection of $H$ with the center of ${\rm GL}_m(E)$ is compact. In particular, this applies to $H={\rm U}_m(E)$.

For the inductive step, let us assume that the theorem holds good for $\Pi = {\rm St}(\pi,n-1)$, so that our goal is to prove it for $\Pi = {\rm St}(\pi,n)$. Recall the exact sequence from Lemma 5.5:

(5.10)\begin{equation} 0 \rightarrow K_n \rightarrow \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) \rightarrow {\rm St}(\pi, n) \rightarrow 0, \end{equation}

with $K_n$ an irreducible representation of ${\rm GL}_{mn}(E)$. Observe that

(5.11)\begin{equation} {\rm Ext}^i_{{\rm U}(V)}( \nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1) , \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1. \end{equation}

Indeed, by Lemma 5.6(ii), one knows the vanishing of ${\rm Ext}^i, i \geq 0$ for the subquotient of the principal series representation $\nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1)$ of ${\rm GL}_{mn}(E)$ supported on a non-open orbit. For the open orbits, the vanishing of ${\rm Ext}^i, i \geq 1$ is consequence of the induction hypothesis and the Kunneth theorem.

Equipped with this vanishing of ${\rm Ext}^i_{{\rm U}(V)}( \nu ^{-(n-1)/2}\pi \times \nu ^{1/2} {\rm St}(\pi, n-1) , \omega _{V, \psi, \mu })$ for all $i \geq 1,$ the usual long exact sequence of Ext groups associated to the short exact sequence of modules in (5.10) gives us isomorphisms:

(5.12)\begin{equation} {\rm Ext}^{i+1}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) \cong {\rm Ext}^i_{{\rm U}(V)}( K_n , \omega_{V, \psi, \mu})\quad \text{for all}\ i \geq 1. \end{equation}

Next, we use the pinned outer automorphism $\phi$ on ${\rm GL}_{mn}(E)$ which is a conjugate of the automorphism $g \rightarrow {}^tg^{-1}$ by a (longest) Weyl group element. The outer automorphism $\phi$ takes standard parabolic subgroups to standard parabolic subgroups and, in particular, takes the parabolic $P_{m,m(n-1)}$ to $P_{m(n-1),m}$. Moreover, for an element $(g_1,g_2)\in {\rm GL}_m(E) \times {\rm GL}_{m(n-1)}(E)$ in the Levi subgroup ${\rm GL}_m(E) \times {\rm GL}_{m(n-1)}(E)$ of $P_{m,m(n-1)}$, one has

\begin{align*} \phi(g_1, g_2) = ({}^tg_2^{-1}, {}^tg_1^{-1}) \in {\rm GL}_{m(n-1)}(E) \times {\rm GL}_m(E). \end{align*}

Applying $\phi$ to the exact sequence (5.10) above, we obtain

\begin{align*} 0 \rightarrow K^\phi_n \rightarrow [\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1)]^\phi \rightarrow {\rm St}(\pi, n)^\phi \rightarrow 0. \end{align*}

By transport of structure,

\begin{align*} [\nu^{-(n-1)/2}\pi \times \nu^{1/2} {\rm St}(\pi, n-1)]^\phi \cong [\nu^{1/2}{\rm St}(\pi, n-1)]^\phi \times [\nu^{-(n-1)/2}\pi]^\phi. \end{align*}

Now by a well-known theorem of Gelfand–Kazhdan, the action of $\phi$ on any irreducible representation of ${\rm GL}_{mn}(E)$ is just the contragredient. Thus, we obtain the exact sequence

\begin{align*} 0 \rightarrow K^\vee_n \rightarrow \nu^{-1/2} {\rm St}(\pi, n-1)^\vee \times (\nu^{(n-1)/2}\pi^\vee) \rightarrow {\rm St}(\pi, n)^\vee \rightarrow 0. \end{align*}

Taking the contragredient of this exact sequence, we get

(5.13)\begin{equation} 0 \rightarrow {\rm St}(\pi,n) \rightarrow \nu^{1/2}{\rm St}(\pi, n-1) \times (\nu^{-(n-1)/2}\pi) \rightarrow K_n \rightarrow 0. \end{equation}

Once again, we have

(5.14)\begin{equation} {\rm Ext}^i_{{\rm U}(V)}( \nu^{1/2} {\rm St}(\pi, n-1) \times \nu^{-(n-1)/2}\pi, \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1. \end{equation}

As for (5.11), this follows by Lemma 5.6(iii), which gives the vanishing of ${\rm Ext}^i$ ($i \geq 0$) for the subquotient of the principal series $\nu ^{1/2} {\rm St}(\pi, n-1) \times \nu ^{-(n-1)/2}\pi$ supported on a non-open orbit, and the vanishing of ${\rm Ext}^i$ ($i \geq 1$) for the open orbits is a consequence of the induction hypothesis and the Kunneth theorem.

Equipped with this vanishing of ${\rm Ext}^i_{{\rm U}(V)}( \nu ^{1/2} {\rm St}(\pi, n-1) \times \nu ^{-(n-1)/2}\pi, \omega _{V, \psi, \mu })$ for all $i \geq 1,$ the usual long exact sequence of Ext groups associated to the short exact sequence of modules in (5.13) gives us isomorphisms:

(5.15)\begin{equation} {\rm Ext}^{i}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) \cong {\rm Ext}^{i+1}_{{\rm U}(V)}( K_n , \omega_{V, \psi, \mu})\quad \text{for all}\ i \geq 1. \end{equation}

Using the isomorphisms in (5.12) and (5.15), we get

\begin{align*} {\rm Ext}^{i}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) \cong {\rm Ext}^{i+2}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu})\quad \text{for all}\ i \geq 1. \end{align*}

Now for any two smooth representations $\pi _1,\pi _2$ of a reductive $p$-adic group $G(F)$, one has

\begin{align*} {\rm Ext}^i_G[\pi_1,\pi_2]=0 \quad \text{for any $i> $ the $F$-rank of $G$.} \end{align*}

Hence, we deduce by (5.15) that

\begin{align*} {\rm Ext}^{i}_{{\rm U}(V)}( {\rm St}(\pi, n) , \omega_{V, \psi, \mu}) = 0\quad \text{for all}\ i \geq 1, \end{align*}

completing the proof of the theorem.

6. Archimedean case

In this section, we consider the archimedean case, so that ${\rm GL}(V) = {\rm GL}_n(\mathbb {C})$. As mentioned before, Conjecture 2.1 in the archimedean case does not determine the unique skew-Hermitian space $V$ which has non-zero contribution. In this section, we shall explain how the conjecture can be refined in the archimedean case to give a definitive answer.

Recall that Hermitian forms over $\mathbb {C}$ are classified by their signatures $(p, q)$. Since skew-Hermitian forms can be obtained from Hermitian ones by scaling by $i$, we shall likewise say that a skew-Hermitian space has signature $(p,q)$ if it has $p$ many $i$ and $q$ many $(-i)$ in an orthogonal basis. We will denote the corresponding space as $V_{p,q}$ and its isometry group as ${\rm U}(V_{p,q}) = {\rm U}_{p,q}$. In particular, in rank one, the two skew-Hermitian forms are classified by their determinant, which is $i$ or $-i$.

An irreducible generic representation $\Pi$ of ${\rm GL}_n(\mathbb {C})$ is an irreducible principal series representation:

\begin{align*} \Pi = {\rm Ind}_{B(\mathbb{C})}^{{\rm GL}_n(\mathbb{C})} (\chi_1 \otimes \cdots \otimes \chi_n) \quad \text{(normalized induction)} \end{align*}

where the $\chi _j$ are characters of $\mathbb {C}^{\times }$. We may write $\chi _j$ as

\begin{align*} \chi_j(z) = |z|^{r_j} \cdot \big( \overline{z}/ z \big)^{k_j/2} \end{align*}

where $k_j \in \mathbb {Z}$.

As in the previous section, we may consider the restriction of the representation $\Pi$ of ${\rm GL}_n(\mathbb {C})$ to a subgroup ${\rm U}(V_{p,q}) = {\rm U}_{p,q} \subset {\rm GL}_n(\mathbb {C})$ by Mackey theory. The open ${\rm U}(V_{p,q})$-orbits on the flag variety ${\rm GL}_n(\mathbb {C})/B$ are associated, as in the $p$-adic case, to the ordered collection of orthogonal (non-degenerate) skew-Hermitian lines $\mathcal {L} = \{ L_1,\ldots,L_n\}$, with $\bigoplus _j L_j \cong V_{p,q}$ as skew-Hermitian spaces. This means that $p$ of the lines $L_i$ have determinant $i$ and the rest have determinant $-i$; we shall call such $\mathcal {L}$ to be $V_{p,q}$-relevant. In particular, the number of open ${\rm U}(V_{p,q})$-orbits is $\binom {n}{p}$.

If we assume that the analog of Theorem 4.8 holds in the archimedean case, then the proof of Corollary 5.3 gives

(*)$$[ {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{V, \psi,\mu}) \cong \bigoplus_{\mathcal{L} } \bigotimes_j {\rm Hom}_{{\rm U}(L_j)}( \chi_j, \omega_{L_j,\psi, \mu}), ]$$

where the sum is taken over those $\mathcal {L}$ which are $V_{p,q}$-relevant. For each $i$, one may apply Theorem 3.1 [Moe87, Rog92]:

\begin{align*} {\rm Hom}_{{\rm U}(L_j)}( \chi_j, \omega_{L_j,\psi, \mu}) \ne 0 \Longleftrightarrow \epsilon(1/2, \chi_j/ \overline{\chi_j} \cdot \mu^{-1}, \psi_E) \cdot \chi_j(-1) = \mu(\det(L_j)), \end{align*}

which shows that at most one $\mathcal {L}$ can have a non-zero contribution to the sum in ($*$). Now let us explicate this local root number condition.

The conjugate-symplectic character $\mu$ of $\mathbb {C}^{\times }$ has the form

\begin{align*} \mu(z) =\bigg( \frac{\bar{z}}{z} \bigg)^{\alpha} \quad \text{with $\alpha \in \frac{1}{2} \mathbb{Z} \setminus \mathbb{Z}$.} \end{align*}

Observe that

\begin{align*} \mu(i) = i^{-2 \alpha}. \end{align*}

Then writing $\chi$ in place of $\chi _j$ for simplicity,

\begin{align*} \chi/ \overline{\chi} \cdot \mu^{-1} : z \mapsto \bigg(\frac{\bar{z}}{z} \bigg)^{k - \alpha}. \end{align*}

Hence, if $\psi$ is the additive character of $\mathbb {R}$ given by

\begin{align*} \psi(x) = e^{2 \pi i x}, \end{align*}

then by [Tat79, 3.2.5] (see also [GGP12a, Proposition 2.1])

\begin{align*} \epsilon(1/2, \chi/ \overline{\chi} \cdot \mu^{-1}, \psi({\rm Tr})) = {\rm sign}(k - \alpha) \cdot i^{2k - 2 \alpha} = {\rm sign}(k - \alpha) \cdot (-1)^k \cdot i^{-2 \alpha}. \end{align*}

Hence, we conclude that

\begin{align*} {\rm Hom}_{{\rm U}(L_j)}( \chi_j, \omega_{L_j, \psi, \mu}) \ne 0 &\Longleftrightarrow \mu(\det(L_j)) = {\rm sign}(k_j - \alpha) \cdot i^{-2 \alpha}, \\ &\Longleftrightarrow \det(L_j) = {\rm sign}(k_j - \alpha) \cdot i. \end{align*}

For this to hold with $\mathcal {L}$ being $V_{p,q}$-relevant, we need

\begin{align*} \#\{ j: k_j > \alpha\} = p \quad \text{and} \quad \# \{j: k_j < \alpha \} = q = n-p. \end{align*}

Hence, our refinement of Conjecture 2.1 in the archimedean case is as follows.

Conjecture 6.1 Assume that $E/F = \mathbb {C} / \mathbb {R}$. Let

\begin{align*} \Pi = {\rm Ind}_{B(\mathbb{C})}^{{\rm GL}_n(\mathbb{C})} (\chi_1 \otimes \cdots \otimes \chi_n) \end{align*}

be an irreducible generic principal series representation of ${\rm GL}_n(\mathbb {C})$ with

\begin{align*} \chi_j(z) = |z|^{r_j} \cdot \big( \overline{z}/ z \big)^{k_j/2}, \quad k_j \in \mathbb{Z}, \end{align*}

and let

\begin{align*} \mu(z) =\bigg( \frac{\bar{z}}{z} \bigg)^{\alpha} \quad \text{with $\alpha \in \frac{1}{2} \mathbb{Z} \setminus \mathbb{Z}$.} \end{align*}

Then for $\psi (x) = e^{2 \pi ix}$,

\begin{align*} {\rm Hom}_{{\rm U}(V_{p,q})}(\Pi, \omega_{V_{p,q},\psi,\mu}) \ne 0 \end{align*}

if and only if

\begin{align*} \#\{ j: k_j > \alpha\} = p \quad \text{and} \quad \# \{j : k_j< \alpha \} = q = n-p. \end{align*}

We have essentially proved this conjecture by our open-orbit analysis above, under the hypothesis that Theorem 4.8 holds in the archimedean case. We leave the analysis of non-open orbits and the resulting extension problems to more capable hands.

7. The conjecture for $A$-parameters

In this section, we shall extend Conjecture 2.1 beyond the setting of generic representations to the setting of non-tempered representations of Arthur type, analogous to what we did in [GGP20] for the classical GGP conjectures. We begin with a brief recollection of this non-tempered conjecture from [GGP20].

7.1 Non-tempered GGP and relevance

In [GGP20], we considered the problem of determining

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)} ( \pi_M, \pi_N) = \text{$0$ or $1$ } \end{align*}

where $\pi _M$ and $\pi _N$ are, respectively, irreducible representations of ${\rm GL}_{n+1}(F)$ and ${\rm GL}_n(F)$ of Arthur type, with associated $A$-parameters

\begin{align*} M_A = \bigoplus_{i=1}^k M_i \boxtimes {\rm Sym}^{d_i-1}(\mathbb{C}^2) \quad \text{and} \quad N_A = \bigoplus_{i=1}^l N_i \boxtimes {\rm Sym}^{e_i-1}(\mathbb{C}^2). \end{align*}

Here, $M_i$ and $N_i$ are irreducible bounded admissible representations of the Weil–Deligne group $WD_F$ and ${\rm Sym}^{d-1}(\mathbb {C}^2)$ is the $d$-dimensional irreducible representation of ${\rm SL}_2(\mathbb {C})$ (the Arthur ${\rm SL}_2(\mathbb {C})$), so that $M_A$ and $N_A$ are representations of $WD_F \times {\rm SL}_2(\mathbb {C})$ of dimension $n+1$ and $n$, respectively. The associated $A$-packets are singletons, containing the irreducible unitary principal series representations:

\begin{align*} \pi_M = \times_{i=1}^r {\rm Sp}( \pi_{M_i}, d_i) \quad \text{and} \quad \pi_N = \times_{i=1}^{l} {\rm Sp}(\pi_{N_i}, e_i), \end{align*}

where $\pi _{M_i}$ refers to the irreducible representation of the appropriate ${\rm GL}$ with $L$-parameter $M_i$ and ${\rm Sp}(\pi _{M_i}, d_i)$ denotes the associated Speh representation (as introduced in § 5.5).

Remark We take this opportunity to correct a misnomer in [GGP20, § 5]. In the first paragraph of [GGP20, p. 2312], the representation with $A$-parameter $M_i \otimes {\rm Sym}^{d_i} (\mathbb {C}^2)$ was denoted by ${\rm Speh}(\pi _{M_i}, d_i)$. Though just a naming convention, it is more customary to denote this representation by ${\rm Speh}(\pi _{M_i}, d_i+1)$. We have followed the latter convention here.

Now the main conjecture in [GGP20] (for the general linear groups) is that

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)} ( \pi_M, \pi_N) = 1 \end{align*}

if and only if the pair $(M_A, N_A)$ is a relevant pair of $A$-parameters. This conjecture has now been proven by Chan [Cha22]. Our goal here is to recall the key notion of ‘relevance’ and make a couple of remarks about it, especially in the context of classical groups.

Definition 7.1 Given two $A$-parameters (of arbitrary dimensions) of ${\rm GL}$-groups

\begin{align*} M_A = \bigoplus_{i=0}^d M_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2) \quad \text{and} \quad N_A = \bigoplus_{i=0}^d N_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2), \end{align*}

we say that $(M_A, N_A)$ is a relevant pair if we have a decomposition of the respective representations of $WD_F$ as

\begin{align*} M_i = M_i^+ + M_i^- \quad \text{and} \quad N_i = N_i^+ + N_i^- \end{align*}

with the property that

\begin{align*} M_i^+ = N_{i+1}^- \quad \text{for $i \geq 0$}\quad \text{and} \quad M_i^- = N_{i-1}^+ \quad \text{for $i \geq 1$.} \end{align*}

This combinatorial definition has a more geometric interpretation which was discussed in [GGP20, § 4].

7.2 Relevance for classical groups

We make a few remarks on the relevance condition for classical groups, clarifying [GGP20].

$\bullet$ The first point is minor but worth noting. The typical GGP conjecture (in the context of ${\rm GL}_n \times {\rm GL}_{n+1}$ say) is formulated as the branching problem of determining

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)^{\Delta}} (\pi_M \otimes \pi_N, \mathbb{C}), \quad \text{rather than } \quad \dim {\rm Hom}_{{\rm GL}_n(F)} ( \pi_M, \pi_N). \end{align*}

When formulated in this way, the non-tempered GGP conjecture would then say that

\begin{align*} \dim {\rm Hom}_{{\rm GL}_n(F)^{\Delta}} (\pi_M \otimes \pi_N, \mathbb{C}) = 1 \end{align*}

if and only if $(M_A, N_A^{\vee })$ is relevant, where $N_A^{\vee }$ is the dual representation of $N_A$.

$\bullet$ Second, in [GGP20, § 6], we formulated the non-tempered GGP conjecture for the classical groups, asserting that the same ‘relevance’ condition plays a crucial role. We take this opportunity to explicate the relevance notion here.

For classical groups, the branching problem concerns the determination of

\begin{align*} \dim {\rm Hom}_H(\pi, \nu), \end{align*}

where $\pi$ is an irreducible representation of

\begin{align*} G= G_1 \times G_2 = {\rm U}_n \times {\rm U}_m \quad \text{(say),} \end{align*}

with $n\geq m$,

\begin{align*} H= {\rm U}_m\ltimes N \subset G \end{align*}

is a subgroup with unipotent radical $N$ and $\nu$ is a certain small representation of $H$. More precisely, $\nu$ is a one-dimensional character if $n \not \equiv m \bmod 2$; this case is referred to as the Bessel case for Hermitian spaces. On the other hand, the case when $n \equiv m \bmod 2$ is referred to as the Fourier–Jacobi case for skew-Hermitian spaces; in this case, $\nu$ is a Weil representation.

The $A$-parameters for classical groups are likewise finite-dimensional representations of $WD_E \times {\rm SL}_2(\mathbb {C})$, where $WD_E$ is the Weil–Deligne group of $E$, with appropriate (conjugate)-duality conditions. Suppose we are given $A$-parameters

\begin{align*} M_A = \bigoplus_{i=0}^d M_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2), \end{align*}

\begin{align*} N_A = \bigoplus_{i=0}^d N_i \boxtimes {\rm Sym}^{i}(\mathbb{C}^2), \end{align*}

with $M_i$ and $N_i$ satisfying appropriate (conjugate-)duality conditions. We can now summarize the relevance conditions required in each case as follows.

1. (i) Orthogonal and symplectic groups (both Bessel and Fourier–Jacobi models): an $A$-parameter $M_A \boxtimes N_A$ of $G_1 \times G_2$ is relevant if and only if $M_A= M_A^{\vee }$ and $N_A= N_A^{\vee }$ form a relevant pair in the sense of Definition 7.1 for ${\rm GL}_m \times {\rm GL}_n$.

2. (ii) Hermitian case (Bessel models): an $A$-parameter $M_A \boxtimes N_A$ of $G_1 \times G_2$ is relevant if and only if $M_A^\vee$ and $N_A$ form a relevant pair in the sense of Definition 7.1 for ${\rm GL}_m\times {\rm GL}_n$.

3. (iii) Skew-Hermitian case (Fourier–Jacobi model): in this case, the definition of the Weil representation $\nu$ requires an extra piece of data, namely a character

   \begin{align*} \mu: E^\times \rightarrow \mathbb{C}^\times \quad \text{with $\mu|_{F^\times} = \omega_{E/F}$.} \end{align*}
   An $A$-parameter $M_A \boxtimes N_A$ of $G_1 \times G_2$ is relevant if and only if $\mu \cdot M_A^\vee$ and $N_A$ are relevant in the sense of Definition 7.1 for ${\rm GL}_m \times {\rm GL}_n$.

7.3 Non-tempered twisted GGP

We shall now formulate the extension of the non-tempered GGP conjecture of [GGP20] to the twisted setting considered in this paper. Hence, with $E/F$ a quadratic extension, suppose we have a representation $\pi _M$ of ${\rm GL}(V)$ with associated $A$-parameter $M_A$. The notion of relevance is not immediately obvious in this setting, as in contrast to the situations discussed above, we do not have a pair of $A$-parameters but only a single one. Nonetheless, we have the following result.

Conjecture 7.2 Let $V$ be an $n$-dimensional $E/F$-skew-Hermitian space. Let $\pi$ be an irreducible admissible representation of ${\rm GL}(V)$ with an $A$-parameter (which is an $n$-dimensional representation of $WD_E \times {\rm SL}_2(\mathbb {C})$) of the form

\begin{align*} M_A = \bigoplus_{i=1}^r M_i \boxtimes {\rm Sym}^{d_i}(\mathbb{C}^2), \end{align*}

where $M_i$ is an irreducible $m_i$-dimensional tempered representation of $WD_E$. If

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] \not = 0, \end{align*}

then $M_A$ is a sum of a tempered $A$-parameter (i.e. with ${\rm SL}_2(\mathbb {C})$ acting trivially) and summands of the form

\begin{align*} N_i \boxtimes {\rm Sym}^{d_i}(\mathbb{C}^2) \oplus \mu\cdot N_i^\sigma \boxtimes {\rm Sym}^{d_i-1}(\mathbb{C}^2), \end{align*}

where $d_i \geq 1$, and the $N_i$ are tempered representations of $WD_E$ with $N^\sigma _i$ their conjugate under the action of ${\rm Gal}(E/F)$. Equivalently, the parameters $M_A$ and $\mu \cdot M_A^\sigma$ should be relevant in the sense of [GGP20].

Conversely, if the parameters $M_A$ and $\mu \cdot M_A^\sigma$ are relevant in the sense of [GGP20], then

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] = \mathbb{C} \end{align*}

for exactly one skew-Hermitian space $V$, namely the one determined as in Conjecture 2.1(iii).

We leave it to the reader to verify that when $E = F \times F$ is split, so that $V = V_1 \times V_2$, the relevance condition in Conjecture 7.2 reduces to the one formulated earlier for a representation $\pi _M = \pi _1 \otimes \pi _2$ of ${\rm GL}(V) = {\rm GL}(V_1) \times {\rm GL}(V_2)$.

7.4 Degenerate principal series

The reader may wonder how we are led to the above conjecture. In fact, we are led to the conjecture by considering the branching problem for degenerate principal series representations. Recall that in the previous three sections, we have appealed to Mackey theory computations to study the twisted branching problem for tempered principal series representations and generalized Steinberg representations. As much of the material there is of a general nature, it is natural to apply them to the analogous restriction problem for degenerate principal series representations. The result is given in the following proposition. Note that the degenerate principal series considered below are of Arthur type. Hence, the proposition serves as a motivation and check for Conjecture 7.2.

Proposition 7.3 Let:

• • $n = a+b$, with $0 < a \leq b \in \mathbb {Z}$;

• • $\chi _1,\chi _2: E^\times \rightarrow \mathbb {C}^\times$ be two unitary characters;

• • $V = V_a \oplus V_b$ be an $n$-dimensional $E/F$-skew-Hermitian space, with $\dim V_a = a$;

• • $P = P_{a,b}$ the maximal parabolic subgroup of ${\rm GL}(V)$ stabilizing $V_a$, with Levi factor ${\rm GL}(V_a) \times {\rm GL}(V_b)$;

• • $\pi = \chi _1 \times \chi _2$ be the degenerate principal series representation of ${\rm GL}(V)$ induced from the corresponding one-dimensional character $(\chi _1 \circ \det _{\,V_a}) \otimes (\chi _2 \circ \det _{\,V_b})$ of $P_{a,b}$.

If

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] \not = 0, \end{align*}

then the following hold:

1. (i) $b=a+1$; and

2. (ii) $\chi _1 = \chi _2^\sigma \cdot \mu$ where $\sigma$ is the Galois involution of $E/F$.

Conversely, if $b=a+1$ and $\chi _1 = \chi _2^\sigma \cdot \mu$, then there is exactly one skew-Hermitian structure on $V$ such that

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi,\mu}] = \mathbb{C}, \end{align*}

and for the other skew-Hermitian space $V'$,

\begin{align*} {\rm Hom}_{{\rm U}(V')}[ \pi, \omega_{V',\psi,\mu}] = 0. \end{align*}

Proof. We shall apply the results from Mackey theory obtained in Proposition 4.4. Recall that the orbits for the action of ${\rm U}(V)$ on $X= {\rm GL}_n(E)/P_{a,b}$ are given by Lemma 4.2. For an $a$-dimensional subspace $X \subset V$ with $\dim (X \cap X^\perp )=d$ with the corresponding subquotient $\pi _X$ of $\pi$, Proposition 4.4 says that

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[\pi_X, \omega_{V,\psi,\mu}] \cong {\rm Ext}^i_{Q/N(Q)}[ (\pi_1)_{d,a-d} \otimes (\pi_2)_{b-d,d} \otimes \delta_{P/Q}^{1/2}, \delta_{Q}^{1/2} \cdot |\!\det\!|^{-1/2} \mu\cdot \omega_{V_{n-2d},\psi,\mu}], \end{align*}

where $Q/N(Q)={\rm GL}_d(E) \times {\rm U}_{a-d}\times {\rm U}_{b-d}$ and the other notation is as given there. Applying this to $\pi _1=\chi _1$ and $\pi _2=\chi _2$, we deduce

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] \cong {\rm Ext}^i_{Q/N(Q)}[ \chi_1 \cdot (\chi_2^\sigma)^{-1} |\!\det\!|^{d/2}, |\!\det\!|^{(n-d)/2} |\!\det\!|^{-1/2}\mu \cdot \omega_{V_{n-2d},\psi,\mu}]. \end{align*}

We shall now study when this Ext group can be non-zero.

Consider first the case when $[X]$ is an open orbit (so that $d = 0$) and $i=0$. In this case each of the $|\!\det\! |$ factors which refers to ${\rm GL}_d(E)$, are trivial for $d=0$, hence it follows by Lemma 5.7 that

\begin{align*} {\rm Hom}_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] = {\rm Hom}_{{\rm U}(V_a)}( \chi_1, \omega_{V_a, \psi,\mu}) \otimes {\rm Hom}_{{\rm U}(V_b)}(\chi_2, \omega_{V_b, \psi,\mu}) = 0. \end{align*}

On the other hand, when $d > 0$, it follows by Lemma 4.6 (on matching powers of $|\!\det\! |$ for the two arguments) that a necessary condition for the non-vanishing of the above Ext group is

\begin{align*} 2d+1=n. \end{align*}

Since $d\leq a \leq b < n=(a+b)$, this implies that we must have

\begin{align*} d=a \quad \text{and} \quad b=a+1, \end{align*}

which means that $[X]$ is the unique closed orbit of ${\rm U}(V)$ on ${\rm GL}_n(E)/P_{a,b}$. In particular, $\pi _X$ is a quotient of $\pi$.

With $a$, $b$ and $d$ related as above, the Ext group in question is

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] \cong {\rm Ext}^i_{{\rm GL}_a(E) \times {\rm U}_1} [ \chi_1 \cdot (\chi_2^\sigma)^{-1}, \mu \cdot \omega_{V_{1},\psi,\mu}], \end{align*}

where $\omega _{V_{1},\psi,\mu }$ is a Weil representation of ${\rm U}(V_1)= {\rm U}_1$ for the one-dimensional skew-Hermitian space $V_1$ with ${\rm disc}(V_1) = {\rm disc}(V)$, and we are regarding $\omega _{V_{1},\psi,\mu }$ as a representation of ${\rm GL}_a(E) \times {\rm U}_1$. Thus, we see that

\begin{align*} \chi_1 = \chi_2^\sigma \cdot \mu \end{align*}

is a necessary condition for the non-vanishing of this Ext group. When this condition holds, the above Ext group becomes ${\rm Ext}^i_{{\rm U}(V_1)}( 1, \omega _{V_1, \psi, \mu })$ and this vanishes if $i > 0$ (since ${\rm U}(V_1)$ is compact).

We have, thus, shown that

\begin{align*} {\rm Hom}_{{\rm U}(V)}( \chi_1 \times \chi_2, \omega_{V, \psi,\mu}) = {\rm Hom}_{{\rm U}(V)}( \pi_X, \omega_{V, \psi,\mu}) \end{align*}

for $[X]$ the unique closed ${\rm U}(V)$-orbit on ${\rm GL}(V) / P_{a,b}$, and a necessary condition for the non-vanishing of this Hom space is

\begin{align*} b = a+1 \quad \text{and} \quad \chi_1 = \chi_2^\sigma \cdot \mu. \end{align*}

In other words, we have proved the first assertion of the proposition.

For the converse, since $[X]$ is the closed orbit of ${\rm U}(V)$ on ${\rm GL}(V) / P_{a,b}$, we have seen that when the above conditions hold, one has

\begin{align*} {\rm Hom}_{{\rm U}(V)}( \chi_1 \times \chi_2, \omega_{V, \psi,\mu}) = {\rm Hom}_{{\rm U}(V)}( \pi_X, \omega_{V, \psi,\mu}) \cong {\rm Hom}_{{\rm U}(V_1)}( 1, \omega_{V_1, \psi, \mu}). \end{align*}

One is thus reduced to the $n=1$ case of Conjecture 2.1 which is known.

The proof of the proposition is now complete.

Remark 7.4 The proof above also proves that for the degenerate principal series representation $\pi = \chi _1\times \chi _2$ of ${\rm GL}_n(E)$, with $b = a+1$ and $\chi _1 = \chi _2^\sigma \cdot \mu$,

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi|_{{\rm GL}_n(E)}, \omega_{V,\psi,\mu}] \cong \sum_{i=j+k} {\rm Ext}^j_{{\rm U}(V_a)}( \chi_1, \omega_{V_a, \psi,\mu}) \otimes {\rm Ext}^k_{{\rm U}(V_b)}(\chi_2, \omega_{V_b, \psi,\mu}). \end{align*}

8. When $E \ne K$: local case

In this section, we consider the general twisted variant of the GGP problem, where $E \ne K$ are two distinct quadratic extensions of a local field $F$. In particular, $F$ is necessarily non-archimedean and we fix a non-trivial additive character $\psi$ of $F$. This case is considerably more intricate and, like the GGP problem, we will need to make use of the local Langlands correspondence for unitary groups to formulate our conjectural answers.

8.1 Biquadratic extension

Let $L = E \otimes _F K$, so that $L$ is a biquadratic extension of $F$. We thus have the following picture.

In particular, we have set

\begin{align*} {\rm Gal}(E/F)\cong {\rm Gal}(L/K) = \langle \sigma \rangle \quad \text{and} \quad {\rm Gal}(K/F) \cong {\rm Gal}(L/E) = \langle \tau \rangle. \end{align*}

Observe that the biquadratic field $L$ contains a third quadratic subfield $E'$ which is the fixed field of $\sigma \cdot \tau$. This field $E'$ will play a role later on.

8.2 Skew-Hermitian spaces

We consider the two isomorphism classes of skew-Hermitian spaces $V$ and $V'$ over $E$ of dimension $n$, and make the following observation.

Lemma 8.1 The two skew-Hermitian spaces $V_K = V \otimes _F K$ and $V'_K = V' \otimes _F K$ are isomorphic over $L$. When $n$ is even, $V_K \cong V'_K$ is the maximally split skew-Hermitian space; whereas when $n$ is odd, $V_K \cong V'_K$ is characterized as the unique skew-Hermitian space whose determinant can be represented by elements of $E^{\times }_0$. In either case, ${\rm U}(V_K) \cong {\rm U}(V'_K)$ is a quasi-split group.

Proof. It suffices to show that $\det V$ and $\det V'$ belong to the same $N_{L/K}(L^{\times })$-coset, when viewed as elements of $K^{\times }$ or $L_0^{\times}$. Since $\det V$ and $\det V'$ belong to the same $F^{\times }$-coset, it suffices to observe that $F^{\times } \subset N_{L/K}(L^{\times })$. Indeed, since $L$ is a biquadratic extension of $F$, $\omega _{L/K} = \omega _{E/F} \circ N_{K/F}$. Hence,

\begin{gather*} \omega_{L/K}(F^{\times}) = \omega_{E/F} \big( N_{K/F}(F^{\times}) \big) = \omega_{E/F}(F^{\times 2}) =1. \end{gather*}

In view of the lemma, we may regard ${\rm U}(V)$ and ${\rm U}(V')$ as subgroups of a fixed ${\rm U}(V_K) = {\rm U}(V'_K)$.

8.3 Local Langlands correspondence

Now we recall the local Langlands correspondence for ${\rm U}(V_K)$. An $L$-parameter for ${\rm U}(V_K)$ is a conjugate-dual $n$-dimensional semisimple representation $M$ of the Weil–Deligne group $WD_L = W_L \times {\rm SL}_2(\mathbb {C})$ of sign $(-1)^{n-1}$. We have studied such conjugate-dual representations in some detail in [GGP12b] and described their associated component groups $A_M$. More precisely, we may write

\begin{align*} M = \bigoplus_{i\in I} V_i \otimes M_i \oplus P \oplus {}^\sigma P^\vee \end{align*}

with $M_i$ distinct conjugate-dual representations of sign $(-1)^{n-1}$, $V_i$ its multiplicity space and $P$ contains all the irreducible summands which are either non-conjugate-dual or conjugate-dual of sign $(-1)^n$, with ${}^\sigma P^\vee$ its conjugate-dual. As we discussed in [GGP12b, § 4], the centralizer group of the $L$-parameter is of the form

\begin{align*} C_M = \prod_{i \in I} {\rm O}(V_i) \times (\text{a connected reductive group}). \end{align*}

Hence, the component group of $C_M$ is an elementary abelian 2-group

\begin{align*} A_M = \prod_{i \in I} \mathbb{Z}/2\mathbb{Z} \cdot a_i, \end{align*}

equipped with a canonical basis indexed by $I$. The element $-1_M$ gives rise to the element

\begin{align*} \sum_{i \in I} \dim(V_i) \cdot a_i \in A_M, \end{align*}

which generates a subgroup of order $\leq 2$ in $A_M$. Now the local Langlands correspondence for ${\rm U}(V_K)$ gives a partition

\begin{align*} {\rm Irr}({\rm U}(V_K)) = \bigsqcup_M \Pi_M, \end{align*}

of ${\rm Irr}({\rm U}(V_K))$ into the disjoint union of finite subsets, the $L$-packets, with the sum running over $L$-parameters of ${\rm U}(V_K)$. Moreover, since we are at the moment concerned only with the quasi-split group ${\rm U}(V_K)$, for each parameter $M$ of ${\rm U}(V_K)$, one has a bijection

\begin{align*} J: \Pi_M \longleftrightarrow {\rm Irr}(A_M / \langle -1_M \rangle). \end{align*}

Here the bijection $J$ is canonical when $n$ is odd and depends on the choice of an equivalence class of Whittaker datum for ${\rm U}(V_K)$ when $n$ is even. In that case, we have seen in [GGP12b] that the equivalence classes of Whittaker data are parameterized by additive characters of $K$ modulo the translation action of $N_{L/K}(L^\times )$. We shall use the Whittaker datum associated to $\psi _K = \psi \circ {\rm Tr}_{K/F}$.

Recall that an $L$-parameter $M$ is generic if the adjoint $L$-factor $L(s, M, {\rm Ad})$ is holomorphic at $s =1$. In that case, the $L$-packet $\Pi _M$ contains a unique representation which is generic with respect to the Whittaker datum associated to $\psi \circ {\rm Tr}_{L/K}$. This representation corresponds to the trivial character of $A_M$ under the bijection $J$.

8.4 Asai factors

We recall from [GGP12b] the notion of Asai $L$-factors and $\epsilon$-factors associated to a representation $M$ of $WD_L$ relative to the quadratic extension $L/E$. If $\tau$ denotes the non-trivial element of ${\rm Aut}(L/E) \cong {\rm Aut}(K/F)$, the representation $M \otimes M^{\tau }$ is $\tau$-invariant and, hence, we have a decomposition

\begin{align*} {\rm Ind}_{WD_L}^{WD_E} (M \otimes M^{\tau}) = {\rm As}_{L/E}^+(M) \oplus {\rm As}_{L/E}^-(M) \end{align*}

of $WD_E$-modules, with ${\rm As}_{L/E}^{\pm }(M) \cong M \otimes M^{\tau }$ as $WD_L$-modules. On ${\rm As}_{L/E}^+(M)$, an element $s \in W_E \setminus W_L$ acts by $v \otimes w \mapsto w \otimes s^2 \cdot v$, whereas on ${\rm As}_{L/E}^-(M)$, this action is twisted by the non-trivial character of $W_E/W_L$ (see [GGP12b, pp. 26–27]), thus ${\rm As}_{L/E}^-(M)= {\rm As}_{L/E}^+(M) \cdot \omega _{L/E}$.

We record here some useful properties of the functor ${\rm As}^{\pm }_{L/E}$. Later we will deal exclusively with ${\rm As}^+_{L/E}$, dropping the sign $+$.

Lemma 8.2 One has the following.

1. (a) If $M = \bigoplus _i M_i$, then

   \begin{align*} {\rm As}_{L/E}^{\epsilon}(M) = \bigoplus_i {\rm As}_{L/E}^{\epsilon}(M_i) \oplus \bigoplus_{i < j} {\rm Ind}_L^E (M_i \otimes M_j^{\tau}). \end{align*}

2. (b) One has ${\rm As}^{\epsilon }_{L/E}(M)^{\vee } \cong {\rm As}^{\epsilon }_{L/E}(M^{\vee })$, where $M^{\vee }$ denotes the dual of $M$.

3. (c) One has ${\rm As}^{\epsilon }_{L/E}(M_1\otimes M_2) \cong {\rm As}^{\epsilon }_{L/E}(M_1) \otimes {\rm As}^{\epsilon }_{L/E}(M_2)$.

4. (d) If $\dim M =1$, in which case $M$ is treated as a character of $WD_L^{ab} = L^{\times }$, ${\rm As}^+_{L/E}(M)$ is the restriction of $M$ from $L^\times$ to $E^{\times }$.

5. (e) As a character of $WD^{ab}_E \cong E^{\times }$,

   \begin{align*} \det({\rm As}^+_{L/E}(M)) = {\rm As}^+(\det(M))^n \cdot \omega_{L/E}^{n(n-1)/2} = \det(M)|^n_{E^{\times}}\cdot \omega_{L/E}^{n(n-1)/2}, \end{align*}
   where $n = \dim M$.

6. (f) If $M$ is an $L$-parameter of ${\rm U}(V_K)$ and, hence, is conjugate-dual (with respect to $L/K$) of sign $(-1)^{n-1}$, then ${\rm As}^{\pm }_{L/E}(M)$ is necessarily conjugate-orthogonal relative to $E/F$.

8.5 Conjectures

We now come to the restriction problem to be studied. For each of the two skew-Hermitian spaces $V$ over $E$, we have the Weil representation $\omega _{V, \psi,\mu }$, where $\mu$ is a conjugate-symplectic character of $E^{\times }$. Then we are interested in determining

\begin{align*} m_V(\pi, \mu) := \dim {\rm Hom}_{{\rm U}(V)} (\pi, \omega_{V,\psi,\mu}) \quad \text{for $\pi \in {\rm Irr}({\rm U}(V_K))$.} \end{align*}

Here is our main local conjecture for arbitrary separable quadratic extensions $E,K$ of $F$, subsuming the earlier Conjecture 2.1 (for the case $E=K$).

Conjecture 8.3

1. (i) For each $\pi \in {\rm Irr}({\rm U}(V_K))$,

   \begin{align*} m_V(\pi, \mu) = \dim {\rm Hom}_{{\rm U}(V)}(\pi, \omega_{V,\psi, \mu}) \leq 1. \end{align*}

2. (ii) Let $M$ be a generic $L$-parameter of ${\rm U}(V_K\!)$ with associated $L$-packet $\Pi _M \subset {\rm Irr}({\rm U}(V_K))$. Then

   \begin{align*} \sum_V \sum_{\pi \in \Pi_M} m_V(\pi, \mu) =1 \end{align*}
   where the first sum runs over the two skew-Hermitian spaces over $E$ of dimension $n$ and the second runs over the $L$-packet $\Pi _M$.

3. (iii) The unique $V_0$ which has non-zero contribution to the sum in part (ii) is characterized by

   \begin{align*} \mu(\det(V_0)) = \epsilon( 1/2, {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) \cdot \det ({\rm As}_{L/E}(M))(e) \cdot \omega_{K/F}(e^2)^{n(n-1)/2}, \end{align*}
   where $e$ is any non-zero trace 0 element of $E$, so that $E = F(e)$.

4. (iv) The unique $\pi \in \Pi _M$ which has non-zero contribution to the sum in part (ii) corresponds via the bijection $J$ to the character of the local component group $A_M = \prod _{i \in I} \mathbb {Z}/2\mathbb {Z} \cdot a_i$ given by

   \begin{align*} \chi(a_i) & = \epsilon( 1/2, {\rm Ind}_L^E( M_i^{\tau} \otimes (M/M_i)) \cdot \mu^{-1}, \psi_{E,e}) \\ & = \epsilon(1/2, [{\rm As}(M_i) + {\rm As}(M) + {\rm As}(M/M_i)] \cdot \mu^{-1}, \psi_{E,e}), \end{align*}
   where $\psi _{E,e}$ is the additive character of $E/F$ defined by $\psi _{E,e}(x) = \psi ({\rm Tr}(ex))$.

We make a few remarks on the above conjecture.

(a) In part (iii), the proposed expression for $\mu (\det (V_0))$ is independent of the choice of the trace 0 element $e$. Moreover, using property (d) in § 8.4 and the fact that $\omega _{L/E}(e) = \omega _{K/F}(N_{E/F}(e)) = \omega _{K/F}(-e^2)$, the equation in part (iii) can be explicated as

\begin{align*} \mu(\det(V_0)) = \epsilon( 1/2, {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) \cdot \det(M) (e)^n \cdot \omega_{K/F}(-1)^{n(n-1)/2}. \end{align*}

Though this may be more compact, our original expression has the advantage that it can be specialized to all possible situations for the pair $(E,K)$, as we shall explain below.

(b) In part (iii), observe that if $E = F(e)$ and $K = F(k)$ with $k \in K^{\times }$ a trace zero element, then

\begin{align*} \omega_{K/F}(e^2) = (k^2, e^2)_F. \end{align*}

In particular, we see that this term only appears when $K$ and $E$ are both fields (as we are assuming in the conjecture).

(c) The distinguished character $\chi$ in part (iv) is indeed trivial on the image of $-1_M$ in $A_M$. Moreover, it is independent of the choice of the trace $0$ element $e$. This follows from the fact that $({\rm As}(M_i) + {\rm As}(M) + {\rm As}(M/M_i)) \cdot \mu ^{-1}$ is an even-dimensional conjugate-symplectic representation of $WD_E$ and hence its determinant is conjugate-orthogonal.

(d) For the skew-Hermitian case considered in [GGP12b], we had defined a distinguished character $\chi$ of the local component group which gives the unique representation in the $L$-packet with non-zero branching multiplicity. This distinguished character automatically picks out the skew-Hermitian space $V_0$ over $E$ which supports the non-zero multiplicity, so that part (iii) is a consequence of part (iv) in the original GGP setting. In the case here, the distinguished character $\chi$ in part (iv) gives a representation of ${\rm U}(V_K)$, but does not specify the $E$-space $V_0$. This is why condition (iii) is needed.

8.6 Specializations

Though we are assuming that $E \ne K$ are distinct quadratic fields in this section, the formulas in Conjecture 8.3(iii) and (iv) make sense for general $(E, K)$. For this, we need to explain how the $L$-parameter of $\Pi \in {\rm Irr}({\rm U}(V_K))$ gives rise to a representation of $WD_L$ and how to interpret the Asai lift relative to $L/E$ in the various situations.

$\bullet$ $E = K$ is a field: this is the setting of § 2. In this case, $L = E \otimes K$ is isomorphic to $E \times E = K \times K$. Note however that the embeddings of $K$ and $E$ into $L$ are different. The embedding of $K$ into $L$ is the diagonal embedding $x \mapsto (x,x)$, whereas that of $E$ into $L$ is $x \mapsto (x, x^{\sigma })$, where ${\rm Aut}(E/F) = \langle \sigma \rangle$. We interpret the Weil–Deligne group of $L$ as $WD_L = WD_K \times WD_K = WD_E \times WD_E$.

Now given an irreducible representation $\Pi$ of ${\rm U}(V_K) = {\rm GL}(V)$, its $L$-parameter $M$ is an $n$-dimensional representation of $W_K = W_E$ and this gives rise to the pair $(M, M^{\vee })$ which we interpret as a representation of $WD_L$. Now the non-trivial element of ${\rm Aut}(L/E)$ acts on $L = E \times E$ via $(x,y) \mapsto (y^{\sigma }, x^{\sigma })$. Thus, its induced action on the representations of $WD_L$ is $(M , M^{\vee }) \mapsto ({}^\sigma \!M ^\vee, M^{\sigma })$ (the switch, followed by the action of $\sigma$). We interpret the Asai lift as the tensor product representation $M \otimes {}^\sigma \!M ^\vee$ of $WD_E$. With these interpretations, the formula in Conjecture 8.3(iii) specializes to that in Conjecture 2.1(iii), in view of remark (a) in § 8.5.

The issue addressed by Conjecture 8.3(iv) is not relevant in this case since the $L$-packet of ${\rm U}(V_K) = {\rm GL}(V)$ is a singleton. However, we note that with the above interpretations, the right-hand side of the formula there is equal to $1$.

$\bullet$ $E$ is a field and $K = F \times F$, so that $L = K \otimes E = E \times E$ and $WD_L = WD_E \times WD_E$: this is the original GGP situation. Then ${\rm U}(V_K) \cong {\rm U}(V) \times {\rm U}(V)$ and an irreducible representation of ${\rm U}(V_K)$ is of the form $\pi _1 \boxtimes \pi _2$ with $\pi _i \in {\rm Irr}({\rm U}(V))$. The $L$-parameters of $\pi _1$ and $\pi _2$ are conjugate-dual representations $M_1$ and $M_2$ of $WD_E$ of sign $(-1)^{n-1}$, giving a representation $(M_1, M_2)$ of $WD_L$. Now since $E$ is embedded diagonally in $L = E \times E$, the non-trivial automorphism of $L/E$ is the switch of the two factors of $E$ in $L$. The Asai lift of $M_1 \boxtimes M_2$ from $L$ to $E$ is interpreted as the internal tensor product $M_1 \otimes M_2$. With these interpretations, the formula in Conjecture 8.3(iii) reads

\begin{align*} \mu(\det(V_0)) = \epsilon(1/2, M_1 \otimes M_2 \otimes \mu^{-1}, \psi_E) \cdot \det(M_1 \otimes M_2)(e). \end{align*}

We leave it to the reader to verify that this reduces to the relevant conjecture in [GGP12b].

$\bullet$ Compared with the other cases, a peculiarity of the original GGP situation is that ${\rm U}(V_K)$ and ${\rm U}(V'_K)$ are not isomorphic when $V$ and $V'$ are the two distinct skew-Hermitian spaces over $E$. Hence, one needs to choose and fix a quasi-split ${\rm U}(V_K)$ to formulate the LLC, before one can consider Conjecture 8.3(iv). When $\dim V$ is even, this choice is unique, but when $\dim V$ is odd, this amounts to choosing a trace zero element $e_0 \in E^{\times }$ (the determinant of the distinguished $V$). Moreover, it is no longer the case that the character given in Conjecture 8.3(iv) is independent of $e$ when $\dim V$ is odd (though it is still the case when $\dim V$ is even). Thus, in Conjecture 8.3(iv), one needs to use the distinguished $e_0$ in the definition of the character $\chi$ when $\dim V$ is odd. With this caveat, we leave it to the reader to verify that the formula for the character $\chi$ in part (iv) specializes to the one we had in [GGP12b].

$\bullet$ $E = F \times F$ and $K$ is a field, so that $L = K \times K$. Here, ${\rm U}(V) = {\rm GL}(V)$ and ${\rm U}(V_K) = {\rm GL}(V_K)$. Given an irreducible generic representation $\Pi$ of ${\rm GL}(V_K)$, and a conjugate-dual character $\mu = (\nu, \nu ^{-1})$ of $E^{\times } / F^{\times } = (F^{\times } \times F^{\times })/F^{\times }$, the multiplicity $\dim {\rm Hom}_{{\rm GL}(V)}(\Pi, \omega _{V,\psi,\mu })$ should be always non-zero. So we expect the proposed identity in Conjecture 8.3(iii) to always hold, after appropriate interpretations.

Now the $L$-parameter of $\Pi$ is an $n$-dimensional representation $M$ of $WD_K$. This gives rise to the pair $(M, M^{\vee })$ which we regard as a representation of $WD_L = WD_K \times WD_K$. The non-trivial automorphism of $L/E$ is the componentwise action of $\tau \in {\rm Aut}(K/F)$ on $L = K \times K$, so the Asai lift from $L$ to $E$ is the pair $({\rm As}_{K/F}(M), {\rm As}_{K/F}(M^{\vee }))$, regarded as a representation of $WD_E = WD_F \times WD_F$. In this case,

\begin{align*} &\epsilon(1/2, {\rm As}_{L/E}(M, M^{\vee}) \cdot\mu^{-1}, \psi_E) \\ &\quad = \epsilon(1/2, {\rm As}_{K/F}(M)\cdot \nu^{-1}, \psi) \cdot \epsilon(1/2, {\rm As}_{K/F}(M)^{\vee} \cdot \nu, \psi). \\ &\quad = \det({\rm As}_{K/F}(M))(-1) \cdot \nu(-1)^n. \end{align*}

Moreover, an element $e \in E = F \times F$ of trace $0$ is of the form $(a, -a)$ for $a \in F^{\times }$. Hence,

\begin{align*} \det({\rm As}_{L/E}(M,M^{\vee})) (e) &= \det({\rm As}_{K/F}(M))(a) \cdot \det({\rm As}_{K/F}(M))(-a)^{-1} \\ &= \det({\rm As}_{K/F}(M))(-1) \end{align*}

and

\begin{align*} \omega_{K/F}(e^2) = \omega_{K/F}(a^2) =1. \end{align*}

Thus, the right-hand side of the formula in part (iii) is $\nu (-1)^n$, which is equal to the left-hand side.

There is also the case where $E = K = F \times F$, which we will leave to the reader. The main reason for formulating Conjecture 8.3 in a uniform way which allows for specialization to the various cases is that in the global setting to be considered in § 11, any one of these local scenarios will arise.

9. Low-rank evidence: $E \ne K$

Just as for Conjecture 2.1, we provide here some evidence for Conjecture 8.3 in low-rank cases. In particular, we shall show the following.

Theorem 9.1 Conjecture 8.3 holds when $\dim V \leq 2$.

The rest of this section is devoted to the verification of the theorem.

9.1 Rank-one case

Assume first that $V$ is a skew-Hermitian space of dimension 1, so that ${\rm U}(V) = E_1\subset {\rm U}(V_K) = L_1$, where $L_1$ denotes the subgroup of elements $x \in L^{\times }$ with $N_{L/K}(x) =1$. Given a character $\chi$ of $L_1$, choose an extension $\tilde {\chi }$ of $\chi$ to $L^{\times }$. Then the $L$-parameter of $\chi$ is the one-dimensional conjugate-orthogonal representation $M = \tilde {\chi }/ \tilde {\chi }^{\sigma }$ of $W_L$. By the theorem of Moen and Rogawski, we know that

\begin{align*} {\rm Hom}_{E_1}(\chi, \omega_{V, \psi, \mu}) \ne 0 \Longleftrightarrow \epsilon(1/2, ( \tilde{\chi}/ \tilde{\chi}^{\sigma})|_{E^{\times}} \otimes \mu^{-1}, \psi_E) \cdot \chi(-1) = \mu(\det(V)). \end{align*}

The local root number above can be written as

\begin{align*} \epsilon(1/2, {\rm As}_{L/E}(M) \cdot \mu^{-1}, \psi_E), \end{align*}

whereas

\begin{align*} \det({\rm As}(M))(e) = \tilde{\chi}(e) / \tilde{\chi}(e^{\sigma}) = \chi(-1). \end{align*}

This shows Conjecture 8.3 when $n= \dim V =1$.

9.2 Rank-two case

Suppose now that $\dim V =2$. In this case, we need to verify the independent statements (iii) and (iv) of Conjecture 8.3. As we have noted before, $V = V_B$ is associated with a quaternion $F$-algebra $B$, with

\begin{align*} {\rm GU}(V_B)\cong (B^{\times} \times E^{\times} ) / \Delta F^{\times}. \end{align*}

The embedding ${\rm GU}(V_B) \hookrightarrow {\rm GU}(V_{B,K})$ is the natural embedding

\begin{align*} (B^{\times} \times E^{\times}) / \Delta F^{\times} \hookrightarrow ((B \otimes_F K)^{\times} \times L^{\times} ) / \Delta K^{\times}, \end{align*}

with $B \otimes _F K \cong M_2(K)$.

A generic $L$-packet of ${\rm U}(V_K)$ is thus given by an irreducible representation

\begin{align*} \Pi \boxtimes \chi \quad \text{ of ${\rm GL}_2(K) \times L^{\times}$,} \end{align*}

with $\omega _{\Pi } \cdot \chi |_{K^{\times }} =1$. If $P$ is the $L$-parameter of $\Pi$, then the $L$-parameter of the corresponding $L$-packet of ${\rm U}(V_K)$ is the conjugate-symplectic (relative to $L/K$) representation

\begin{align*} M =P|_{WD_L} \otimes \chi \end{align*}

of $WD_L$. On the other hand, the Weil representation $\omega _{\psi,\mu, B}[ \chi |_{E^{\times }}]$ of ${\rm U}(V_B)$ is an irreducible summand of the representation

\begin{align*} \Sigma_{B,N} \otimes \chi \end{align*}

of $B^{\times } \times E^{\times }$ restricted to $(B^{\times })^+ \times E^{\times }$, where as in § 3.2, $\Sigma _{B,N}$ has $L$-parameter

\begin{align*} N = {\rm Ind}_E^F (\mu \cdot\chi|_{E^\times}^{-1} ). \end{align*}

The corresponding $L$-parameter of ${\rm U}(V_B)$ is the conjugate-symplectic (relative to $E/F$) representation

\begin{align*} N|_{WD_E} \otimes \chi|_{E^{\times}}. \end{align*}

Now we consider the sum

(9.2)\begin{equation} \sum_{\pi \in \Pi_M} \dim {\rm Hom}_{{\rm U}(V_B)}( \pi, \omega_{\psi,\mu, B}). \end{equation}

Via the above identifications, one sees that this sum is simply

\begin{align*} \dim {\rm Hom}_{(B^{\times})^+} (\Pi, \omega_{\psi,\mu, B}) = \dim {\rm Hom}_{B^{\times}}( \Pi, \Sigma_{B,N}). \end{align*}

In other words, we are reduced to a twisted trilinear form problem as in § 3.2. Hence, by a result of the third author, cf. [Pra92], this dimension is at most $1$ and is non-zero if and only if

(9.3)\begin{equation} \epsilon(1/2, {\rm As}_{K/F}(P) \otimes {\rm Ind}_E^F (\mu^{-1} \cdot \chi|_{E^{\times}}), \psi) \cdot \omega_{K/F}(-1) = \mu(\det(V_B)). \end{equation}

Now the local root number can be explicated as

\begin{align*} \epsilon(1/2, {\rm As}_{K/F}(P) \otimes {\rm Ind}_E^F (\mu^{-1} \cdot \chi|_{E^{\times}}), \psi) &= \epsilon(1/2, {\rm Ind}_E^F ({\rm As}_{K/F}(P)|_{WD_E} \cdot \chi|_{E^{\times}} \cdot \mu^{-1}),\psi), \\ &= \epsilon(1/2, {\rm Ind}_E^F ({\rm As}_{L/E}( P|_{WD_L} \cdot \chi) \cdot \mu^{-1}), \psi), \\ &= \epsilon(1/2, {\rm As}_{L/E}(P|_{WD_L} \cdot \chi) \cdot \mu^{-1}, \psi_E), \\ &= \epsilon(1/2, {\rm As}_{L/E}(M) \cdot \mu^{-1},\psi_E). \end{align*}

In the above, we have used the facts that

\begin{align*} {\rm As}_{K/F}(P) |_{WD_E} \cong {\rm As}_{L/E} \big( P|_{WD_L} \big) \end{align*}

and

\begin{align*} {\rm As}_{L/E}(P|_{WD_L}) \otimes \chi |_{E^{\times}} = {\rm As}_{L/E}( P|_{WD_L} \otimes \chi) = {\rm As}_{L/E}(M). \end{align*}

On the other hand, with $n =2$,

\begin{align*} \det({\rm As}_{L/E}(M))(e)^n \cdot \omega_{K/F}(e^2)^{n(n-1)/2} = \det(M)(e)^2 \cdot \omega_{L/E}(e) \cdot \omega_{K/F}(e^2) = \omega_{K/F}(-1), \end{align*}

since $\det (M)$ is conjugate-orthogonal and, hence, trivial on $e^2 \in F^{\times }$, and $\omega _{L/E}(e) = \omega _{K/F}(-e^2)$. Hence, the equality (9.3) is precisely the statement of Conjecture 8.3(iii).

We now come to Conjecture 8.3(iv). Continuing with the analysis above, let us fix $V = V_B$ such that (9.3) holds, so that the sum in (9.2) is equal to $1$, and we need to determine which element in the $L$-packet $\Pi _M$ has non-zero contribution. Now the members of the $L$-packet are given by the restriction of $\Pi$ to ${\rm GL}_2(K)^+$. If this restriction is irreducible, then we leave it to the readers to convince themselves that Conjecture 8.3(iv) holds. Let us examine the more intricate case when this restriction is the sum of two irreducible summands, i.e. when $\Pi$ is dihedral with respect to $L/K$. Thus, we see that the problem at hand is a refined version of the twisted trilinear form problem, relative to the embedding ${\rm GL}_2(F) \subset {\rm GL}_2(K)^+$.

Since $\Pi$ is dihedral with respect to $L/K$, $P|_{WD_L}$ is reducible and so is $M = P|_{WD_L} \cdot \chi$. To understand the $L$-packet, we shall return to the setting of unitary groups, as $\Pi _M$ can be constructed via theta lifting from rank-one skew-Hermitian spaces.

9.3 Unitary theta lifts

Let $M = M_1 + M_2$ be an $L$-parameter of ${\rm U}(V_B)(K)$ with $M_1$ and $M_2$ conjugate-symplectic characters of $W_L$. The $L$-packet $\Pi _M$ has two representations of ${\rm U}(V_B)(K)$, which we may denote by $\pi ^+$ and $\pi ^{-}$ (these are $\pi ^{++}, \pi ^{--}$ of [GGP12a]), so that $\pi ^+$ is generic with respect to the Whittaker datum determined by $\psi _K = \psi \circ {\rm Tr}_{K/F}$. Note that by Lemma 8.1, ${\rm U}(V_B)(K)$ is always the quasi-split unitary group in two variables, so the representations on the anisotropic form of ${\rm U}(V_B)(K)$ does not arise in our considerations. We shall explain how these representations $\pi ^{\pm }$ can be constructed as theta lifts from ${\rm U}_1$.

Let $W^{\pm }$ be the two rank-one Hermitian spaces over $L$ with $\omega _{L/K}({\rm disc}(W^{\pm })) = \pm 1$. In particular, the Hermitian form on $W^+$ is $(x,y)\mapsto x \cdot y^{\tau }$, with ${\rm Gal}(L/K) = \langle \tau \rangle$. Then for $\epsilon = \pm 1$, ${\rm U}(W^{\epsilon }) \times {\rm U}(V_K)$ is a reductive dual pair where $V_K = V_B \otimes _F K$. Likewise, we may consider the rank-two Hermitian space

\begin{align*} W^{\epsilon}_E := {\rm Res}_{L/E}(W^{\epsilon}) \quad \text{with Hermitian form ${\rm Tr}_{L/E}( -, -)_{W^{\epsilon}}$.} \end{align*}

This rank-two Hermitian space over $E$ has discriminant

\begin{align*} {\rm disc}(W^{\epsilon}_E) = N_{K/F}( k \cdot {\rm disc}(W^{\epsilon})) \in F^{\times}/ N(E^{\times}), \end{align*}

where $k \in K^{\times }$ is any trace 0 element; we leave the verification of this to the reader. Then ${\rm U}(W^\epsilon _E) \times {\rm U}(V)$ is a reductive dual pair, and we have the following seesaw diagram.

To consider the theta correspondences for these two dual pairs, we need to select splitting characters in each case, and to obtain a seesaw identity from the seesaw diagram, we need to select these two sets of splitting characters compatibly. With the goal of obtaining the $L$-packet $\Pi _M$ of ${\rm U}(V_K)$ as theta lifts from ${\rm U}(W^{\pm })$, we shall select these splitting characters as follows.

• • Recall that $M_1$ is a conjugate symplectic character of $L^{\times }$ relative to $L/K$. Then its restriction $M_1|_{E^{\times }}$ is a conjugate-orthogonal character of $E^{\times }$ relative to $E/F$ (because $F^{\times } \subset N_{L/K}(L^{\times })$).

• • For the equal rank dual pair ${\rm U}(V) \times {\rm U}(W_E)$ over $F$, we use the pair of splitting characters $(M_1|_{E^{\times }}, M_1|_{E^{\times }})$, and the additive character $\psi$ of $F$.

• • For the almost equal rank dual pair ${\rm U}(V_K\!) \times {\rm U}(W^{\epsilon })$ over $K$, we use the pair $(M_1, M_1 \circ N_{L/E}) = (M_1, M_1 \cdot M_1^{\tau })$ and the character $\psi _K$ of $K$.

With these splitting characters and additive characters fixed, one can consider the associated theta correspondences for the two dual pairs. Moreover, one has the seesaw identity associated to the above seesaw diagram. For this, one needs to specify the irreducible representations one starts with on ${\rm U}(W^{\epsilon })$ and ${\rm U}(V)$.

1. (i) For the dual pair ${\rm U}(W^{\epsilon }) \times {\rm U}(V_K)$, if one starts with the character $\chi _{M_1^{\tau }M_2}$ of ${\rm U}(W^{\epsilon })$ with $L$-parameter $M_1^{\tau } \cdot M_2$, then its theta lift to ${\rm U}(V_K)$ has $L$-parameter $M =M_1 +M_2$. As $\epsilon$ varies over $\pm$, the two representations so obtained are the elements $\pi ^{\epsilon }$ of the $L$-packet $\Pi _M$.

2. (ii) For the dual pair ${\rm U}(V) \times {\rm U}(W_E)$, we start with the Weil representation $\omega _{\psi, \mu, V}[\chi _{M_1M_2}]$ of ${\rm U}(V)$ whose central character is the character $\chi _{M_1M_2}$ of $E_1$ with $L$-parameter $M_1M_2$ and whose $L$-parameter is $N = \mu + \mu ^{-1} M_1M_2$. Its theta lift to ${\rm U}(W_E)$, if non-zero, has the same $L$-parameter.

From the seesaw identity, we see that

\begin{align*} {\rm Hom}_{{\rm U}(V)}( \pi^{\epsilon}, \omega_{\psi,\mu, V}) \cong {\rm Hom}_{{\rm U}(W^{\epsilon})}(\Theta(\omega_{\psi, \mu, V}[\chi_{M_1M_2}]), \chi_{M_1^{\tau}M_2}), \end{align*}

so that

\begin{align*} {\rm Hom}_{{\rm U}(V)}(\pi^{\epsilon}, \omega_{\psi,\mu, V}) \ne 0 \Longrightarrow \Theta(\omega_{\psi, \mu, V}[\chi_{M_1M_2}]) \ne 0. \end{align*}

By the theta dichotomy theorem [HKS96, GI16], the latter holds if and only if

\begin{align*} \omega_{E/F}(- k^2) \cdot \epsilon = \omega_{E/F}({\rm disc}(W_E^{\epsilon})) = \epsilon(1/2, N \cdot M_1|_{E^{\times}}^{-1}, \psi_{E,e}) \cdot \mu(\det(V)). \end{align*}

The local root number on the right-hand side is equal to

\begin{align*} &\epsilon(1/2, {\rm As}_{L/E}(M_1)^{-1} \cdot \mu, \psi_{E,e}) \cdot \epsilon(1/2, {\rm As}_{L/E}(M_2) \cdot \mu^{-1}, \psi_{E,e}) \\ &\quad = \epsilon(1/2, {\rm As}_{L/E} (M_1) \cdot \mu^{-1}, \psi_{E, e}) \cdot \epsilon(1/2, {\rm As}_{L/E}(M_2) \cdot \mu^{-1}, \psi_{E,e}) \cdot \omega_{E/F}(-1) . \end{align*}

On the other hand, by Conjecture 8.3(iii), which we have demonstrated above, we know that

\begin{align*} \mu(\det(V)) &= \epsilon(1/2, {\rm As}_{L/E}(M) \cdot \mu^{-1},\psi_E) \cdot \omega_{K/F}(-1) \\ &= \epsilon(1/2, {\rm As}_{L/E}(M) \cdot \mu^{-1},\psi_{E,e}) \cdot \omega_{K/F}(e^2). \end{align*}

Assembling these together, we see that

\begin{align*} \epsilon & = \epsilon(1/2, [{\rm As}_{L/E} M_1 + {\rm As}_{L/E}(M_2) + {\rm As}_{L/E}(M)] \cdot \mu^{-1}, \psi_{E, e})\cdot \omega_{E/F}(k^2) \cdot \omega_{K/F}(e^2), \\ &= \epsilon(1/2, [{\rm As}_{L/E} M_1 + {\rm As}_{L/E}(M_2) + {\rm As}_{L/E}(M)] \cdot \mu^{-1}, \psi_{E, e}), \end{align*}

as predicted by Conjecture 8.3(iv), where for the second equality, we have used

\begin{align*} \omega_{K/F}(e^2) = (k^2, e^2)_F = \omega_{E/F}(k^2). \end{align*}

Note that by Lemma 8.2(a), the last epsilon factor can be simplified as

\begin{align*} \epsilon = \epsilon(1/2, {\rm Ind}_L^E (M_1^{\tau} \cdot M_2)\cdot \mu^{-1}, \psi_{E,e}). \end{align*}

We have thus completed the proof of Theorem 9.1. For concreteness, we highlight the results obtained for the rank-two case.

Proposition 9.4 Suppose we are given:

• • a quadratic extension $E/F$ of non-archimedean local fields;

• • a quaternion $F$-algebra $B$ with associated skew-Hermitian space $V_B$ of dimension 2 over $E$;

• • a quadratic field extension $K \ne E$ with associated biquadratic field $L = E \otimes K$;

• • an $L$-parameter $M = M_1 + M_2$ of ${\rm U}(V_B)(K)$, with $M_1$ and $M_2$ conjugate-symplectic characters of $W_L$, whose $L$-packet $\Pi _M$ has two representations $\pi ^+$ and $\pi ^{-}$ of ${\rm U}(V_B)(K)$, so that $\pi ^+$ is generic with respect to the Whittaker datum determined by $\psi _K = \psi \circ {\rm Tr}_{K/F}$.

Then one has

\begin{align*} {\rm Hom}_{{\rm U}(V_B)}( \pi^{\epsilon}, \omega_{\psi,\mu, V_B}) \ne 0 \iff \begin{cases} \mu(\det(V_B)) = \epsilon(1/2, {\rm As}_{L/E}(M) \cdot \mu^{-1},\psi_E) \cdot \omega_{K/F}(-1), \\ \epsilon = \epsilon(1/2, {\rm Ind}_L^E (M_1^{\tau} \cdot M_2)\cdot \mu^{-1}, \psi_{E,e}),\end{cases} \end{align*}

where $e \in E^{\times }_0$.

10. Unitary principal series: $E \ne K$

In this section, we shall study the restriction problem for unitary principal series representations and show the analog of Corollary 5.3 in the $E \ne K$ setting. Recall that we have the following diagram of fields and Galois automorphisms.

The biquadratic field $L$ contains a third quadratic subfield $E'$ which is the fixed field of $\sigma \cdot \tau$. Let $V$ be a skew-Hermitian space (relative to $E/F$) of dimension $n$ over $E$ and $V_K= V \otimes _F K$, the corresponding skew-Hermitian space (relative to $L/K$) over $L=KE$. We also let $\tau$ denote the Galois automorphism acting on $V_K$ and ${\rm U}(V_K)$ with fixed points $V$ and ${\rm U}(V)$, respectively.

10.1 Mackey theory

We shall consider the restriction to ${\rm U}(V)$ of a parabolically induced representation from a maximal parabolic subgroup of ${\rm U}(V_K)$. The following theorem is an analog of Theorem 4.8.

Theorem 10.1 Let $V_K$ be the $n$-dimensional skew-Hermitian space relative to $L/K$ which is the base change of any $n$-dimensional skew-Hermitian space relative to $E/F$.

• • Let $P = MN$ be a maximal parabolic subgroup of ${\rm U}(V_K)$ which is the stabilizer of an $a$-dimensional isotropic subspace of $V_K$, with Levi factor

  \begin{align*} M \cong {\rm GL}_a(L) \times {\rm U}_{n-2a}(K). \end{align*}

• • Let $\pi = \pi _1 \rtimes \pi _2 = {\rm Ind}_P^{{\rm U}(V_K)} (\pi _1 \otimes \pi _2)$ be a tempered principal series representation of ${\rm U}(V_K)$, with $\pi _1 \in {\rm Irr}({\rm GL}_a(L))$ and $\pi _2 \in {\rm Irr}({\rm U}_{n-2a}(K))$.

For any skew-Hermitian $V$ relative to $E/F$ such that $V \otimes _F K \cong V_K$, let $\omega _{V,\psi, \mu }$ be a Weil representation of ${\rm U}(V)$.

Then for all $i \geq 0$,

\begin{align*} & \sum_{V} {\rm Ext}^i_{{\rm U}(V)}[ \pi, \omega_{V,\psi, \mu}]\\ &\quad \stackrel{(1)} = \sum_{i=j+k} \bigg(\sum_{V'_a} {\rm Ext}^j_{{\rm U}(V'_a)}[ \pi_1, \omega_{V'_a,\psi, \mu \circ N_{L/E}}] \bigg) \otimes \bigg(\sum_{V_{n-2a}} {\rm Ext}^k_{{\rm U}(V_{n-2a})}[ \pi_2, \omega_{V_{n-2a},\psi, \mu}] \bigg), \end{align*}

where:

• • the sum over $V$ runs over the two skew-Hermitian spaces relative to $E/F$ of dimension $n$;

• • the sum over $V'_a$ runs over the two skew-Hermitian spaces relative to $L/E'$ of dimension $a$;

• • the sum over $V_{n-2a}$ runs over the two skew-Hermitian spaces relative to $E/F$ of dimension $n-2a$;

• • $\omega _{V'_a,\psi, \mu \circ N_{L/E}}$ and $\omega _{V_{n-2a},\psi, \mu }$ denote the corresponding Weil representations of ${\rm U}(V'_a)$ and ${\rm U}(V_{n-2a})$.

In particular, for $i=0$,

\begin{align*} & \sum_{V} {\rm Hom}_{{\rm U}(V)}[ \pi, \omega_{V,\psi, \mu}]\\ &\quad \stackrel{(2)} = \bigg(\sum_{V'_a} {\rm Hom}_{{\rm U}(V'_a)}[ \pi_1, \omega_{V'_a,\psi, \mu \circ N_{L/E} }] \bigg)\otimes \bigg(\sum_{V_{n-2a}} {\rm Hom}_{{\rm U}(V_{n-2a})}[ \pi_2, \omega_{V_{n-2a},\psi, \mu}] \bigg). \end{align*}

The isomorphisms in both of the above equations (1) and (2) come from the open orbits. More precisely, if $[X]$ is a non-open orbit of ${\rm U}(V)$ on ${\rm U}(V_K)/P$, contributing (by the Mackey theory) a certain representation $\pi _X$ of ${\rm U}(V)$ as a subquotient of $\pi$, then

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi, \mu}] = 0, \end{align*}

for all $i \geq 0$.

Proof. The proof of this theorem is almost identical to the corresponding theorem for the $E=K$ case, i.e. Theorem 4.8, so we will be brief. It again depends on using the Mackey theory to calculate the representation $\pi _X$ of ${\rm U}(V)$ as a subquotient of $\pi$ supported on each orbit $[X]$ of ${\rm U}(V)$ on the partial flag variety ${\rm U}(V_K)/P$. Hence, we first investigate the orbits of ${\rm U}(V)$ on ${\rm U}(V_K)/P$, and their associated stabilizers in ${\rm U}(V)$.

The partial flag variety ${\rm U}(V_K)/P$ parameterizes $a$-dimensional isotropic $L$-subspaces $X$ of $V_K$. Since $\tau$ acts on $V_K$, we have an action $X \mapsto X^{\tau }$ of $\tau$ on $U(V_K)/P$. For each isotropic $X$, let $P_X \subset {\rm U}(V_K\!)$ be the stabilizer of $X$ in ${\rm U}(V_K)$, so that $P_X = M_X N_X$ is a maximal parabolic subgroup with Levi factor

\begin{align*} M_X = {\rm GL}(X) \times {\rm U}(X^{\perp}/X). \end{align*}

Let

\begin{align*} Q_X = {\rm U}(V) \cap P_X \end{align*}

be the stabilizer of $X$ in ${\rm U}(V)$, with $N_{Q_X}$ its unipotent radical. Therefore, $Q_X$ preserves the flag:

\begin{align*} 0 \subset X \cap X^\tau \subset X \subset X^\perp \subset ( X \cap X^\tau)^\perp \subset V_K, \end{align*}

and there is a natural map

\begin{align*} Q_X \rightarrow P_X \rightarrow M_X = {\rm GL}(X) \times {\rm U}(X^\perp/X). \end{align*}

Hence, a representation $\pi _1 \boxtimes \pi _2$ of $M_X = {\rm GL}(X) \times {\rm U}(X^\perp /X)$ gives rise by pullback to a representation of $Q_X$, which can then be induced to ${\rm U}(V)$ to obtain the representation $\pi _X$ of ${\rm U}(V)$ supported on the ${\rm U}(V)$-orbit of $X$.

After the above generalities, we now consider different cases according to the types of $X$.

Case 1: $X \cap X^\tau \not = 0$. Set $d = \dim_L (X \cap X^{\tau})$. The space $X \cap X^\tau$ is defined over $E$, so let $Y\subset V$ be such that $Y_L := Y \otimes _E L= X \cap X^\tau$. The space $Y$ is isotropic, and it is easy to see that $Q_X$ has the following properties.

1. (1) $Q_X$ is a subgroup of the parabolic subgroup

   \[ P_Y = ({\rm GL}(Y) \times {\rm U}(V_0)) \ltimes N_Y \subset {\rm U}(V) \]
   stabilizing $Y$, with $V_0$ a nondegenerate subspace such that $Y^{\perp } = Y \oplus V_0$. Indeed, one has
   \[ Q_X = ({\rm GL}(Y) \times H) \ltimes N_Y \]
   for some subgroup $H \subset {\rm U}(V_0)$. To see this, note that elements of ${\rm GL}(Y) \cdot N_Y$ preserve $Y$ and act as identity on $Y^{\perp }/Y$. Since
   \[ Y_L \subset X \subset Y^{\perp}_L, \]
   we see that ${\rm GL}(Y) \cdot N_Y$ preserves $X$ and hence lies in $P_X \cap {\rm U}(V) = Q_X$.

2. (2) For the natural map

   \[ Q_X \rightarrow {\rm GL}(X) \times {\rm U}(X^\perp/X), \]
   the center $Z_Y$ of $N_Y$ lies in the kernel, since elements of $Z_Y$ act as identity on $Y^{\perp }$.

3. (3) For the composite map

   \[ \pi_X:Q_X \rightarrow {\rm GL}(X) \times {\rm U}(X^{\perp}/X) \rightarrow {\rm GL}(X), \]
   the image of ${\rm GL}(Y) \cdot N_Y$ under $\pi_X$ is contained in the parabolic subgroup
   \[ P_{d, a-d} = ({\rm GL}_d(L) \times {\rm GL}_{a-d}(L)) \rtimes N_{d, a-d} \]
   stabilizing the subspace $Y_L = X \cap X^\tau \subset X$. Indeed, ${\rm GL}(Y)$ is mapped isomorphically to the subgroup ${\rm GL}(Y) \subset {\rm GL}(Y_L) = {\rm GL}_d(L)$ and $N_Y$ is mapped surjectively onto the unipotent radical $N_{d, a-d}$ of $P_{d,a-d}$.

   To see the assertion on surjectivity, suppose we are given an element $g \in N_{d, a-d}$, so that $g \in {\rm GL}(X)$ acts as identity on $Y_L = X \cap X^{\tau }$ and on $X/ Y_L$. Then $g$ extends to a map on $X + X^{\tau }$, still denoted by $g$. Moreover, this extended map $g$ preserves the skew-Hermitian structure on $X+ X^{\tau }$; this is because the image of $g - 1$ lies in $Y_L$, which is orthogonal to $X + X^{\tau }$. Now note that the space $X + X^{\tau }$ and the map $g \in {\rm GL}(X + X^{\tau })$ are both defined over $E$. By Witt's theorem, we can thus find an element $\tilde {g} \in {\rm U}(V)$ such that $\tilde {g}$ induces $g$ on $X + X^{\tau }$ and hence stabilizes $X$. Such a $\tilde {g}$ thus belongs to $Q_X$ and is sent to $g$ under the composite map here.

4. (4) The composite map

   \[ Q_X \rightarrow {\rm GL}(X) \times {\rm U}(X^{\perp}/X) \rightarrow {\rm U}(X^{\perp}/X) \]
   is trivial on ${\rm GL}(Y)$.

As in the proof of Theorem 4.8, these properties and Lemma 4.7 imply that

\begin{align*} & {\rm Ext}^i_{{\rm U}(V)}(\pi_X, \omega_{V,\psi, \mu}) \\ &\quad \cong {\rm Ext}^i_{{\rm U}(V)}[ {\rm ind}_{Q_X}^{{\rm U}(V)} (\pi_1 \otimes \pi_2 \otimes \delta_{P_X/Q_X}^{1/2}), \omega_{V,\psi, \mu}] \\ &\quad \cong {\rm Ext}^i_{Q_X/Z_Y}[ (\omega_{V,\psi^-, \mu^{-1}})_{Z_Y}, \delta_{Q_X}^{1/2} \cdot (\pi_1 \otimes \pi_2 \otimes \delta_{P_X/Q_X}^{1/2})^\vee ]\\ &\quad \cong {\rm Ext}^i_{Q_X/N_Y}[ (\pi_1)_{d,a-d} \otimes \pi_2 \otimes \delta_{P_X}^{1/2}, \delta_{Q_X}\cdot \mu \cdot |\!\det\!|_E^{-1/2} \cdot \omega_{n-2d,\psi, \mu}] \\ &\quad\cong {\rm Ext}^i_{Q_X/N_Y}(A,B), \end{align*}

where $(\pi _1)_{d,a-d}$ denotes the un-normalized Jacquet module of $\pi _1$ with respect to the parabolic subgroup $P_{d,a-d}$ of ${\rm GL}(X)\cong {\rm GL}_a(L)$ stabilizing $Y_L= X \cap X^{\tau }$.

Now we examine the central characters occurring in $A$ and $B$ as ${\rm GL}(Y)$-modules. A simple computation gives

\[ \delta_{P_X} = |\!\det\!|_L^{n-a}, \quad \delta_{Q_X}= |\!\det\!|_E^{n-d} \quad \text{and} \quad \delta_{P_{d,a-d}} = | \det|_L^{a-d}. \]

Since $\pi _1$ is tempered, one sees by Casselman's criterion that the central exponents of

\[ A = (\pi_1)_{d,a-d} \otimes \pi_2 \otimes \delta_{P_X}^{1/2}, \]

regarded as a representation of ${\rm GL}(Y)$, have the form

\[ |\!\det\!|_L^{({a-d+\epsilon})/{2}} \cdot |\!\det\!|_L^{({n-a})/{2}} = |\!\det\!|_L^{({n-d+\epsilon})/{2}} = |\!\det\!|_E^{n-d+\epsilon} \]

for $\epsilon \geq 0$. On the other hand, the only central exponent occurring in $B$ is

\[ \delta_{Q_X} \cdot |\!\det\!|_E^{-1/2} = |\!\det\!|_E^{n-d -\frac{1}{2}}. \]

Since the central exponents of $A$ and $B$ (regarded as ${\rm GL}(Y)$-modules) are different, we have shown that

\[{\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi,\mu}] = 0 \quad \text{for all $i \geq 0$.} \]

Case 2: $X \cap X^\tau = 0$, but $Z_0= X\cap X^{\tau \perp } \not = 0$. In this case, $X+X^\tau$ is a degenerate skew-Hermitian space defined over $E$ whose nullspace is $Z_0+Z_0^\tau$, i.e.

\begin{align*} Z_0+Z_0^\tau = (X+X^\tau) \cap (X+X^\tau)^\perp. \end{align*}

Let $Z$ be the subspace of $V$ such that $Z\otimes L =Z_0+Z_0^\tau$, so that $Z$ is an isotropic subspace of $V$. In this case, it is easy to see that the subgroup $Q_X$ of ${\rm U}(V)$ preserving $X$ has the following properties.

1. (1) The subgroup $Q_X$ contains the unipotent radical of the parabolic subgroup of ${\rm U}(V)$ stabilizing the isotropic subspace $Z\subset V$.

2. (2) The image of the natural map from $Q_X$ to ${\rm GL}(X)$ given as the composite

   \begin{align*} Q_X \rightarrow {\rm GL}(X) \times {\rm U}(X^\perp/X) \rightarrow {\rm GL}(X) \end{align*}
   lands inside the parabolic subgroup defined by the subspace $Z_0= X\cap X^{\tau \perp } \subset X$, containing the unipotent radical of this parabolic subgroup of ${\rm GL}(X)$, as well as ${\rm GL}(X \cap X^{\tau \perp })$.

A similar analysis as in Case 1 (based on appropriate central character analysis) allows us to conclude that

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi_X, \omega_{V,\psi}] = 0, \end{align*}

for all $i \geq 0$.

Case 3: both $X \cap X^\tau = 0$, and $X\cap X^{\tau \perp } = 0$. In this case, the ${\rm U}(V)$-orbit of $X$ is open. Such isotropic spaces $X \subset V_K$, up to ${\rm U}(V)$-conjugacy, are in bijective correspondence with ${\rm U}(V)$-conjugacy classes of non-degenerate subspaces $W\subset V$ of dimension $2a$, since such a subspace $W$ has, up to ${\rm U}(W)$ conjugacy, a unique subspace $X\subset W_K$ such that $X \cap X^\tau = 0$ and $X\cap X^{\tau \perp } = 0$ (the proof of this is given in Lemma 10.2).

Now let $Q_X$ be the stabilizer of $X$ in ${\rm U}(V)$. The following lemma allows us to determine this stabilizer.

Lemma 10.2 Let $W$ be a $2a$-dimensional non-degenerate skew-Hermitian space over $E$ and let $X \subset W \otimes _F K =W_K$ be an isotropic subspace of $W_K$ such that

\begin{align*} \text{$X \cap X^\tau = 0$ and $ X + X^\tau = W_K$,} \end{align*}

where we recall that ${\rm Gal}(K/F) = \langle \tau \rangle$. Then we have the following.

1. (i) The isotropic subspace $X \subset W_K$ with the above properties is unique up to the action of ${\rm U}(W)$ on $W_K$;

2. (ii) The stabilizer of $X$ in ${\rm U}(W)$ is isomorphic to ${\rm U}(W_X)$, where $W_X$ is the skew-Hermitian space on the underlying vector space $X$ relative to $L/E'$ (for $E'$ the third quadratic field contained in the biquadratic extension $L = E \otimes K$), defined by

   \begin{align*} (x_1,x_2) = \langle x_1, \tau x_2\rangle. \end{align*}

3. (iii) The determinant of the $2a$-dimensional skew-Hermitian space $W$ for $E/F$ and the $a$-dimensional skew-Hermitian space $W_X$ for $L/E'$ are related by (as elements of $F^\times /N_{E/F}( E^\times )$):

   \begin{align*} \det (W) = N_{L/E}[ k^a \det (W_X)] = (-k^2)^a N_{L/E} \det (W_X) , \end{align*}
   where $k$ is any non-zero element of $K$ whose trace to $F$ is zero.

Further, the restriction of the Weil representation $\omega _{W,\psi, \mu }$ of ${\rm U}(W)$ to $U(W_X)$ is the Weil representation of $\omega _{W_X, \psi \circ {\rm Tr}_{E'/F}, \mu \circ N_{L/E}}$.

Proof. (i) Let $X$ and $X'$ be two $L$-vector subspaces of $W_K$ satisfying the properties in the lemma. Let

\begin{align*} \phi: X \longrightarrow X' \end{align*}

be a $L$-linear isomorphism of vector spaces. Then $\phi$ extends uniquely to a $L$-linear automorphism (still denoted by $\phi$) of $X+X^{\tau } = W_K$ defined by

\begin{align*} \phi(x^\tau) = \phi(x)^\tau, \quad \text{for $x \in X$.} \end{align*}

Since this satisfies $\phi \circ \tau = \tau \circ \phi$, it follows by Galois descent that $\phi$ is defined over $E$, i.e. $\phi$ is obtained by base change from an $E$-linear isomorphism

\begin{align*} \phi_0: W \longrightarrow W. \end{align*}

Moreover, one checks by a direct computation that $\phi _0$ preserves the given skew-Hermitian structure on $W$ if and only if $\phi$ is compatible with the $L/E'$-skew-Hermitian structure on $X$ and $X'$ defined in part (ii), i.e. $\phi$ is an isomorphism of $L/E'$-skew-Hermitian spaces:

\begin{align*} \phi: W_X \longrightarrow W_{X'}. \end{align*}

Hence, to show that there is an element of ${\rm U}(W)$ which carries $X$ to $X'$, it remains to show that $W_X$ and $W_{X'}$ are necessarily isomorphic as $L/E'$-skew-Hermitian spaces. We shall show this and hence complete the proof of part (i) only after we demonstrate part (iii), using Lemma 10.3.

(ii) Taking $X' = X$ in part (i) above, one deduces that the stabilizer of $X$ in ${\rm U}(W)$ is precisely ${\rm U}(W_X)$.

(iii) For the $L/E'$-skew-Hermitian space $W_X$ defined in part (ii), there is a natural $E/F$ skew-Hermitian structure on $W_X$ obtained by taking the same vector space as $W_X$, now treated as an $E$-vector space and denoted by $R_E(W_X\!)$, with the skew-Hermitian form which is the $L/E$-trace of the skew-Hermitian form on $W_X$. Define a map $\phi : X\rightarrow W$ by

\begin{align*} \phi(x)=x+ x^\tau \in W \quad \text{ for $x \in X$.} \end{align*}

It is easy to check that $\phi$ induces an isomorphism of the $E/F$ skew-Hermitian spaces $R_E(W_X\!)$ and $W$. Now we appeal to the Lemma 10.3 below to complete the proof of part (iii).

Lemma 10.3 With the quadratic extensions $E,K,E'$ of $F$ as before, let $\mathcal {W}$ be an $L/E'$-skew-Hermitian space with a skew-Hermitian form $\langle -, - \rangle$. Let $R_{E}(\mathcal {W})$ be the same space $\mathcal {W}$ regarded as a vector space over $E$, which comes equipped with a natural $E/F$ skew-Hermitian structure $(-,-)$:

\begin{align*} (w_1,w_2) = \langle w_1, w_2 \rangle + \langle w_1, w_2 \rangle^\tau. \end{align*}

Fix an element $k \in K^{\times }$ with ${\rm Tr}_{K/F} (k) = 0$. Then, with $a = \dim \mathcal {W}$, one has

\begin{align*} k^a \det \mathcal{W} \in E^{\prime\times}, \end{align*}

and

\begin{align*} N_{L/E} (k^a \det \mathcal{W}) = N_{E'/F } (k^a \det \mathcal{W}) = \det R_E(\mathcal{W}), \end{align*}

as elements of $F^\times / N_{E/F}( E^\times )$.

Moreover, if $\mathcal {W}' \ncong \mathcal {W}$ is the other $L/E'$-skew-Hermitian space of the same dimension, then $R_E(\mathcal {W}') \ncong R_E(\mathcal {W}')$.

Proof By writing $\mathcal {W}$ as an orthogonal sum of lines over $L$, we are reduced to proving the lemma for a one-dimensional skew-Hermitian space for $L/E'$ which we take to be the vector space $L$ with the skew-Hermitian structure:

\begin{align*} \langle \ell_1, \ell_2 \rangle = \ell_1 x\ell_2^{\sigma \tau}, \end{align*}

for $x \in L^\times$ with $x+x^{\sigma \tau }=0$.

This gives rise to an $E/F$ skew-Hermitian structure on $L$ by

\begin{align*} (\ell_1, \ell_2) = \langle \ell_1, \ell_2 \rangle + \langle \ell_1, \ell_2 \rangle^\tau = \ell_1 x\ell_2^{\sigma \tau} + \ell_1^\tau x^\tau \ell_2^{\sigma}. \end{align*}

For this $E/F$-skew-Hermitian space $L$, $\{1,k\}$ is a basis, for which the Gram matrix is given by

\begin{align*} A = \bigg( \begin{array}{@{}cc@{}} x+ x^\tau & -k(x-x^\tau) \\ k(x-x^\tau) & -k^2(x+x^\tau) \end{array} \bigg) , \end{align*}

so that

\begin{align*} \det A = -4k^2xx^\tau. \end{align*}

Now since $(kx)^{\sigma \tau }= kx$, we see that $kx$ belongs to $E'^\times$, as desired.

For the final statement, it suffices to show that

\begin{align*} \det(R_E(\mathcal{W}')) \ne \det(R_E(\mathcal{W})) \in F^{\times}/ N_{E/F}(E^{\times}), \end{align*}

or, equivalently, that

\begin{align*} \omega_{E/F} ( \det(R_E(\mathcal{W}')) \cdot \det(R_E(\mathcal{W}))) = -1. \end{align*}

By the identity proved above, this is equivalent to showing that

\begin{align*} ( \omega_{E/F} \circ N_{E'/F} ) \big(\! \det(\mathcal{W}') \cdot \det(\mathcal{W}) \big) = -1. \end{align*}

However, this desired identity holds since

\begin{align*} \omega_{E/F} \circ N_{E'/F} = \omega_{L/E'}. \end{align*}

This completes the proof of Lemma 10.3.

As we mentioned, Lemma 10.3 completes the proof of part (iii). The last assertion in Lemma 10.3 also allows us to complete the proof of part (i). Indeed, in the proof of part (iii), we have shown that $R_E(W_X) \cong W$ as $E/F$-skew-Hermitian spaces. Hence, with $X$ and $X'$ as in the proof of part (i), we deduce that $R_E(W_X) \cong R_E(W_{X'})$. In view of the last assertion in Lemma 10.3, one thus deduces that $W_X \cong W_{X'}$ as $L/E'$-skew-Hermitian spaces.

Finally, we observe that the restriction of the Weil representations made in Lemma 10.2 is the precise version of the well-known assertion that the restriction of a Weil representation of ${\rm Sp}(4n,F)$ to ${\rm Sp}(2n,E')$ takes a Weil representation of ${\rm Sp}(4n,F)$ to a Weil representation of ${\rm Sp}(2n,E')$. This completes the proof of Lemma 10.2.

Applying Lemma 10.2, we find

\begin{align*} Q_X \cong {\rm U}(W_X) \times {\rm U}(W_X^{\perp}) \subset {\rm U}(V) \end{align*}

with $\dim W_X =2a$ and $\dim W_X^{\perp } = n-2a$. We can now conclude the proof as in Theorem 4.8.

The proof of Theorem 10.1 is now complete.

The following proposition is obtained as a corollary to Theorem 10.1.

Proposition 10.4 Let $V$ be an $n$-dimensional skew-Hermitian space relative to $E/F$, and $V_K=V\otimes _F K = V\otimes _EL$ its base change to an $n$-dimensional skew-Hermitian space relative to $L/K$.

• • Let $V_K= X + X^\tau + W'_K$ with $X$ an isotropic subspace of $V_K$ such that $X\cap X^\tau = 0$. Assume that both $(X+X^\tau )$ and $W'_K = W'\otimes _E L$ are defined over $E$, are non-degenerate skew-Hermitian spaces over $E$, and are perpendicular to each other. Let $P = MN$ be a maximal parabolic subgroup of ${\rm U}(V_K)$ which is the stabilizer of $X$, with Levi factor

  \begin{align*} M \cong {\rm GL}(X) \times {\rm U}(W'_K). \end{align*}

• • Let

  \begin{align*} \pi = \pi_1 \rtimes \pi_2 = {\rm Ind}_P^{{\rm U}(V_K)} (\pi_1 \otimes \pi_2) \end{align*}
  be a tempered principal series representation of ${\rm U}(V_K)$, with $\pi _1 \in {\rm Irr}({\rm GL}(X))$ and $\pi _2 \in {\rm Irr}({\rm U}(W'_K))$.

By Lemma 10.2, the vector space $X$ over $L$ carries a natural $L/E'$-skew-Hermitian structure (where $E'$ is the quadratic extension of $F$ inside $L$ different from $E,K$), that we denote by $W$ (so $W$ as a vector space over $L$ is the same as $X$). If Conjecture 8.3(i)–(iii) holds for:

1. (1) the representation $\pi _1$ of ${\rm GL}(X)$ containing the unitary subgroup ${\rm U}(W)$, of size $a$ for the extension $L/E'$;

2. (2) $\pi _2 \in {\rm Irr}({\rm U}(W'_K))$;

then it holds also for the representation $\pi = \pi _1 \rtimes \pi _2$ of ${\rm U}(V_K)$.

Proof. That Conjecture 8.3(i) holds for the representation $\pi = \pi _1 \rtimes \pi _2$ of ${\rm U}(V_K)$ if and only if it does for both the representations $\pi _1$ and $\pi _2$ is the content of our previous theorem.

We will next prove the analogous assertion on Conjecture 8.3(ii). For this, let the representations of the Weil–Deligne group of $L$ associated to $\pi _1,\pi _2$ be $M_1, M_2$. Then the parameter of the representation $\pi$ of ${\rm U}(V_K)$ is $M= M_1 + {^\sigma M_1}^\vee + M_2$. We need to prove that if equations (1) and (2) below hold, then so does equation (3). Here is equation (1):

\begin{align*} & \mu(\det(W')) \\ &\quad = \epsilon( 1/2, {\rm As}_{L/E}(M_2) \otimes \mu^{-1}, \psi_E) \cdot \det ({\rm As}_{L/E}(M_2))(e) \cdot \omega_{K/F}(e^2)^{(n-2a)(n-2a-1)/2} \\ &\quad = \epsilon( 1/2, {\rm As}_{L/E}(M_2) \otimes \mu^{-1}, \psi_E) \cdot \det (M_2)(e)^{n-2a} \cdot \omega_{K/F}(-1)^{(n-2a)(n-2a-1)/2} \\ &\quad \stackrel{(1)}= \epsilon( 1/2, {\rm As}_{L/E}(M_2) \otimes \mu^{-1}, \psi_E) \cdot \det (M_2)(e)^{n} \cdot \omega_{K/F}(-1)^{n(n-1)/2} \cdot \omega_{K/F}(-1)^{a}, \end{align*}

where we have used the observation that $\det M_2$ is a character of $L^\times /K^\times$, hence is trivial on $e^2$. Here is equation (2):

\begin{align*} & \mu(N_{L/E} \det(W_X)) \\ &\quad = \epsilon( 1/2, M_1 \otimes {}^{\sigma \tau} \! M_1 {}^{\vee} \otimes \mu^{-1} \circ N_{L/E}, \psi_L) \cdot \det ( M_1 )(-1)^a \cdot \omega_{L/E'}(-1)^{a(a-1)/2} , \\ &\quad\stackrel{(2)}= \epsilon( 1/2, M_1 \otimes {}^{\sigma \tau} \! M_1 ^{\vee} \otimes \mu^{-1} \circ N_{L/E}, \psi_L) \cdot \det ( M_1 )(-1)^a , \end{align*}

where $\mu ^{-1} \circ N_{L/E}$ denotes the character of $L^\times$ obtained from the character $\mu ^{-1}$ of $E^\times$ by composing with the norm map $N_{L/E}: L^\times \rightarrow E^\times$, and $\psi _L$ is the character of $L$ obtained from the character $\psi _E$ of $E$ obtained by composing with the trace map from $L$ to $E$. Equation (3) is

\begin{align*} & \mu(\det(V)) \\ & \quad= \epsilon( 1/2, {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) \cdot \det ({\rm As}_{L/E}(M))(e) \cdot \omega_{K/F}(e^2)^{n(n-1)/2} \\ & \quad = \epsilon( 1/2, {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) \cdot \det(M)(e)^{n} \omega_{L/E}(e)^{n(n-1)/2} \cdot \omega_{K/F}(e^2)^{n(n-1)/2} \\ &\quad = \epsilon( 1/2, {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) \cdot \det(M)(e)^{n} \cdot \omega_{K/F}(-1)^{n(n-1)/2} \\ &\quad \stackrel{(3)}= \epsilon( 1/2, {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) \cdot \det(M_1)(-1)^n \cdot \det(M_2)(e)^{n} \cdot \omega_{K/F}(-1)^{n(n-1)/2}. \end{align*}

The proof that equations (1) and (2) imply equation (3) depends essentially on relating $\epsilon ( 1/2, {\rm As}_{L/E}(M) \otimes \mu ^{-1}, \psi _E)$ to $\epsilon ( 1/2, {\rm As}_{L/E}(M_2) \otimes \mu ^{-1}, \psi _E)$ and $\epsilon ( 1/2, M_1 \otimes {}^{\sigma } \!M_1 ^{\vee } \otimes \mu ^{-1}\circ N_{L/E}, \psi _L)$, given that $M= M_1 + {}^{\sigma } \!M_1^\vee + M_2$ with ${}^{\sigma } \!M_2^\vee = M_2$. We begin with the following calculation:

\begin{align*} {\rm As}_{L/E}(M) & = {\rm As}_{L/E}(M_1) + {\rm As}_{L/E}({}^{\sigma} \!M_1 ^\vee ) + {\rm As}_{L/E}(M_2) \\ &\quad + {\rm Ind}_L^E( M_1 \otimes M^\tau_2) + {\rm Ind}_L^E( {}^{\sigma} \!M_1^\vee \otimes M^\tau_2) + {\rm Ind}_L^E( M_1 \otimes {}^{\sigma \tau}\! M_1^\vee) \\ & \stackrel{(4)}= {\rm As}_{L/E}(M_1) + {\rm As}_{L/E}({}^{\sigma} \!M_1 ^\vee ) + {\rm Ind}_L^E( M_1 \otimes M^\tau_2) + {\rm Ind}_L^E( {}^{\sigma} M_1^\vee \otimes M^\tau_2) \\ & \quad +{\rm As}_{L/E}(M_2) + {\rm Ind}_L^E( {}^{\sigma \tau}\! M_1^\vee \otimes M_1). \end{align*}

Now, for any representation $N$ of $W_E$, one has

\begin{align*} \epsilon(N+ {}^{\sigma} \!N ^{\vee}, \psi_E) = \det(N)(-1). \end{align*}

Hence, we find (using a calculation on the determinant of the Asai representation ${\rm As}_{L/E}(M_1) \otimes \mu ^{-1}$) that

\begin{align*} \epsilon( [{\rm As}_{L/E}(M_1) + {\rm As}_{L/E}({}^{\sigma} \!M_1 ^\vee )] \otimes \mu^{-1}, \psi_E) & = \det [{\rm As}_{L/E}(M_1)] (-1) \mu^{a^2}(-1) \\ & = \det(M_1)^a(-1) \omega_{L/E}(-1)^{a(a-1)/2} \mu^{a^2}(-1) \\ & \stackrel{(5)}= \det(M_1)^a(-1) \mu^{a}(-1) , \end{align*}

where in the last equality, we have used that $\omega _{L/E}(-1)=1$ since $\omega _{L/E} = \omega _{K/F} \circ N_{E/F}$. Similarly, using that $M_2 \cong {}^{\sigma } \!M_2 ^ \vee$, and a calculation on the determinant of the induced representation ${\rm Ind}_L^E( M_1 \otimes M^\tau _2)$, we find that

\begin{align*} \epsilon ( [{\rm Ind}_L^E( M_1 \otimes M^\tau_2) + {\rm Ind}_L^E( {}^{\sigma} \!M_1^\vee \otimes M^\tau_2)] \otimes \mu^{-1}, \psi_E) &= \det [{\rm Ind}_L^E( M_1 \otimes M^\tau_2) \otimes \mu^{-1}](-1) \\ & = \det ( M_1 \otimes M^\tau_2) (-1), \\ &\stackrel{(6)}= \det ( M_1)(-1)^n, \end{align*}

where, in the second equality, we have used the facts that $\omega _{L/E}(-1)=1$, and $\det M_2(-1) =1$.

By the inductive nature of the epsilon factors for representations of dimension 0, we have

\begin{align*} \epsilon( {\rm Ind}_L^E( {}^{\sigma \tau}\! M_1^\vee \otimes M_1) \otimes \mu^{-1}, \psi_E) & = \epsilon( {}^{\sigma \tau}\! M_1^\vee \otimes M_1 \otimes \mu^{-1}\circ N_{L/E} , \psi_L) \cdot \epsilon(\omega_{L/E}, \psi_E)^{a^2},\\ & = \epsilon( {}^{\sigma \tau}\! M_1^\vee \otimes M_1 \otimes \mu^{-1}\circ N_{L/E} , \psi_L) \cdot \omega_{L/E}(e)^{a},\\ & \stackrel{(7)}= \epsilon( {}^{\sigma \tau}\! M_1^\vee \otimes M_1 \otimes \mu^{-1}\circ N_{L/E} , \psi_L) \cdot \omega_{K/F}(-e^2)^{a}, \end{align*}

From equations (4), (5), (6), and (7), we see that

\begin{align*} \epsilon( {\rm As}_{L/E}(M) \otimes \mu^{-1}, \psi_E) & = \epsilon( [{\rm As}_{L/E}(M_2) + {\rm Ind}_L^E( {}^{\sigma \tau}\! M_1^\vee \otimes M_1)] \otimes \mu^{-1}, \psi_E) \\ & \quad \cdot \det(M_1)^a(-1) \mu^{a}(-1) \cdot \det ( M_1) (-1)^n,\\ & = \epsilon( {\rm As}_{L/E}(M_2)\otimes \mu^{-1}, \psi_E) \epsilon( {}^{\sigma \tau}\! M_1^\vee \otimes M_1 \otimes \mu^{-1}\circ N_{L/E} , \psi_L) \\ & \quad \cdot \det(M_1)^a(-1) \mu^{a}(-1) \cdot \det ( M_1) (-1)^n \omega_{K/F}(-e^2)^{a^2}, \\ & \stackrel{(8)}= \epsilon( {\rm As}_{L/E}(M_2)\otimes \mu^{-1}, \psi_E) \epsilon( {}^{\sigma \tau}\! M_1^\vee \otimes M_1 \otimes \mu^{-1}\circ N_{L/E} , \psi_L) \\ & \quad \cdot \det(M_1)^{n+a}(-1) \mu^{a}(-1) \omega_{K/F}(-e^2)^{a}, \end{align*}

By equation (8), one sees that equations (1) and (2) imply equation (3), using the following identity from Lemma 10.2 of elements of $F^\times /N_{E/F}( E^\times )$,

\begin{align*} (-k^2)^a \det W' \cdot N_{L/E} \det (W) = \det V, \end{align*}

as well as the relation of the characters $\omega _{K/F}$ and $\omega _{E/F}$ to the (quadratic) Hilbert symbol of $F$:

\begin{align*} \omega_{K/F}(x) & = (k^2,x), \\ \omega_{E/F}(x) & = (e^2,x), \end{align*}

which implies

\begin{align*} \omega_{K/F}(e^2) = (k^2,e^2) = \omega_{E/F}(k^2) . \end{align*}

Finally, under the standard identification of the character of component groups under parabolic induction, it is easy to see that the recipe in Conjecture 8.3(iii) holds. This amounts to the identity (for $M_i = {}^{\sigma } \!M_i ^\vee$):

\begin{align*} \epsilon({\rm Ind}_L^E(M_i^\tau \otimes [M_1 + {}^{\sigma} \!M_1^\vee]\otimes \mu^{-1}, \psi_{E,e}) = 1, \end{align*}

which is easy to see.

We have thus finished the proof of Proposition 10.4.

Remark 10.5 The arguments given here also prove that for any tempered representation $\pi$ of ${\rm U}(V_K)$ which is a direct summand of a representation of ${\rm U}(V_K)$ parabolically induced from a unitary cuspidal representation of a Levi subgroup of ${\rm U}(V_K)$,

\begin{align*} {\rm Ext}^i_{{\rm U}(V)}[ \pi, \omega_{V,\psi}] = 0, \quad\text{for all}\ i \geq 1. \end{align*}

This vanishing of higher Ext is as proposed in [Pra18], but is not as precise as Theorem 5.9.

10.2 ${\rm U}(V)$-orbits on the full flag variety

Using Theorem 10.1, and its corollary, Proposition 10.4, one can inductively deduce Conjecture 8.3(i)–(iii) for irreducible unitary principal series representations of ${\rm U}(V_K)$ induced from a Borel subgroup. However, we shall give an alternative treatment involving the analysis of the ${\rm U}(V)$-orbits on the full flag variety of ${\rm U}(V_K)$, which has a rather nice structure that may be of independent interest.

Proposition 10.6 Let:

• • $L = E \otimes K$ be a biquadratic extension and let $E'$ be the third quadratic subfield of $L$;

• • $V$ be a skew-Hermitian space relative to $E/F$ of dimension $n$.

For a skew-Hermitian space $W$ relative to $L/E'$, let ${\rm Res}_{L/E}(W)$ be the same space $W$ regarded as a vector space over $E$ (of twice the dimension) together with the associated $E/F$-skew-Hermitian structure (obtained by taking the trace), so that

\begin{align*} {\rm U}(W) \subset {\rm U}({\rm Res}_{L/E}(W)). \end{align*}

Then we have the following.

(i) If $\dim V = n =2d$ is even, there are $2^{d-1}$ open ${\rm U}(V)$-orbits on the flag variety of ${\rm U}(V_K)$. The open orbits are parameterized by ordered collection of lines

\begin{align*} \mathcal{L} = \{ L_1, L_2,\ldots, L_d \}, \end{align*}

where each $L_i$ is a rank-one skew-Hermitian space relative to $L/E'$, subject to the condition of $V$-relevance:

\begin{align*} \det(V) = \prod_i \det( {\rm Res}_{L/E}(L_i)). \end{align*}

The stabilizer group for the orbit corresponding to $\mathcal {L}$ is

\begin{align*} {\rm U}(\mathcal{L}) = \prod_i {\rm U}(L_i) \subset \prod_i {\rm U}( {\rm Res}_{L/E}( L_i)) \subset {\rm U}(V). \end{align*}

(ii) Suppose that $\dim V = n = 2d+1$ is odd. There are $2^d$ open ${\rm U}(V)$-orbits on the flag variety of ${\rm U}(V_K)$. The open orbits are parameterized by ordered collection

\begin{align*} \mathcal{L} = \{ L_1, L_2,\ldots, L_d ; V_0 \}, \end{align*}

where each $L_i$ is a rank-one skew-Hermitian space relative to $L/E'$, and $V_0$ is a rank-one skew-Hermitian space relative to $E/F$, subject to the condition of $V$-relevance:

\begin{align*} \det(V) = \prod_i \det( {\rm Res}_{L/E}(L_i)) \cdot \det(V_0). \end{align*}

In particular, $V_0$ is determined by $\{ L_1,\ldots, L_d \}$. The stabilizer group associated to $\mathcal {L}$ is

\begin{align*} {\rm U}(\mathcal{L}) = \prod_i {\rm U}(L_i) \times {\rm U}(V_0) \subset \prod_i {\rm U}( {\rm Res}_{L/E}( L_i)) \times {\rm U}(V_0) \subset {\rm U}(V). \end{align*}

10.3 Unitary principal series

Using Proposition 10.6 and Theorem 10.1, we can study the restriction of a unitary principal series

\begin{align*} \Pi = {\rm Ind}_B^{{\rm U}(V_K)} \chi \end{align*}

to ${\rm U}(V)$ and show the following.

Theorem 10.7 Conjecture 8.3(i)–(iii) hold for the tempered $L$-packet consisting of the constituents of the unitary principal series representation $\Pi$.

Proof. The argument is similar to that of Corollary 5.3 and it will be convenient to treat the cases of even or odd $\dim V$ separately. We shall only write down the details for the case of even $\dim V$, leaving the odd case as an exercise for the interested reader.

Assume thus that $\dim V = n = 2d$ is even, so that

\begin{align*} \Pi = {\rm Ind}_B^{{\rm U}(V_K)} (\chi_1 \otimes \cdots \otimes \chi_d) \end{align*}

for some unitary characters $\chi _i$ of $L^{\times }$. By Theorem 10.1, we see that

(10.8)\begin{equation} {\rm Hom}_{{\rm U}(V)}(\Pi, \omega_{\psi,\mu, V}) \cong \bigoplus_{\mathcal{L}} \bigotimes_i {\rm Hom}_{{\rm U}(L_i)}(\chi_i, \omega_{\psi, \mu, L_i^E}), \end{equation}

where the sum runs over $V$-relevant $\mathcal {L}$ and we have written $L_i^E$ for ${\rm Res}_{L/E}(L_i)$.

We thus need to analyze the non-vanishing of ${\rm Hom}_{{\rm U}(L_i)}(\chi _i, \omega _{\psi, \mu, L_i^E})$. As in the $E = K$ case, this comes down to an application of the theorem of Moen and Rogawski, i.e. Theorem 3.1. Indeed, by the functorial property of the Weil representation, the restriction of $\omega _{\psi,\mu, L_i^E}$ of ${\rm U}(L_i)$ is simply the Weil representation $\omega _{\psi _K, \mu \circ N_{L/E}, L_i}$ of ${\rm U}(L_i)$. Hence,

\begin{align*} {\rm Hom}_{{\rm U}(L_i)}(\chi_i, \omega_{\psi, \mu, L_i^E}) \ne 0 \end{align*}

if and only if

(10.9)\begin{equation} \mu( N_{L/E}(\det(L_i))) = \chi_i(-1) \cdot \epsilon(1/2, \chi_i/ \chi_i^{\tau \sigma} \cdot (\mu \circ N_{L/E})^{-1}, \psi_L), \end{equation}

where the local root number is considered over $L$.

Hence, we see that at most one $\mathcal {L}$ in (10.8) has non-zero contribution, and this $\mathcal {L} = \{L_i \}$ is characterized by having (10.9) holding for all $i$. By Lemma 10.3,

\begin{align*} \det(L_i^E) = N_{L/E}( k \cdot \det(L_i)) \in F^{\times}/ N_{E/F}(E^{\times}) \end{align*}

where $k \in K_0^{\times }$. Thus, if ${\rm Hom}_{{\rm U}(V)}(\Pi, \omega _{\psi,\mu, V}) \ne 0$, then

(10.10)\begin{align} \mu(\det(V)) &=\prod_i \mu(\det(L_i^E)) \nonumber\\ &= \prod_i \mu( N_{L/E}(k \cdot \det(L_i)) \nonumber\\ &= \omega_{E/F}(-k^2)^d \cdot \prod_i \chi_i(-1) \cdot \prod_i \epsilon(1/2, \chi_i/ \chi_i^{\tau \sigma} \cdot (\mu \circ N_{L/E})^{-1}, \psi_L), \end{align}

with the last equality following by (10.9).

On the other hand, according to Conjecture 8.3(iii), one should expect that

\begin{align*} \mu(\det(V)) = \epsilon(1/2, {\rm As}_{L/E}(M) \cdot\mu^{-1}, \psi_E) \cdot \det({\rm As}(M))(e) \cdot \omega_{K/F}(e^2)^{n(n-1)/2}, \end{align*}

where

\begin{align*} M = \bigoplus_i M_i = \bigoplus_i (\chi_i + (\chi_i^{\sigma})^{-1}) \end{align*}

is the $L$-parameter of $\Pi$. Let us explicate this and compare it with the expression for $\mu (\det (V))$ in (10.10).

By Lemma 8.2(a),

\begin{align*} {\rm As}_{L/E}(M) = \bigoplus_i {\rm As}_{L/E}(M_i) \oplus \bigoplus_{i< j} {\rm Ind}_L^E (M_i^{\tau} \otimes M_j), \end{align*}

with $M_i = \chi _i + (\chi _i^{\sigma })^{-1}$. Likewise, by Lemma 8.2(a) and (c),

\begin{align*} {\rm As}_{L/E}(M_i) = \chi_i|_{E^{\times}} + (\chi_i^{\sigma})^{-1}|_{E^{\times}} + {\rm Ind}_L^E \chi_i/ \chi_i^{\tau \sigma}, \end{align*}

and it follows that

\begin{align*} &\epsilon(1/2, {\rm As}_{L/E}(M_i) \cdot \mu^{-1}, \psi_E) \\ &\quad = \chi_i(-1) \cdot \omega_{E/F}(-1) \cdot \epsilon(1/2, {\rm Ind}_L^E \chi_i/ \chi_i^{\tau \sigma} \cdot \mu^{-1}, \psi_E) \\ &\quad =\chi_i(-1) \cdot \omega_{E/F}(-1) \cdot \omega_{K/F}(-e^2) \cdot \epsilon(1/2, \chi_i/ \chi_i^{\tau \sigma} \cdot (\mu \circ N_{L/E})^{-1}, \psi_L). \end{align*}

In the above computation, we have repeatedly used the facts:

1. (1) $\epsilon (1/2, N + (N^{\sigma })^{\vee }, \psi _E) =\det (N)(-1);$

2. (2) $\epsilon (1/2, {\rm Ind}_L^E N, \psi _E) = \epsilon (1/2, N, \psi _L) \cdot \epsilon (1/2, \omega _{L/E},\psi _E)^{\dim N};$

3. (3) $\epsilon (1/2, \omega _{L/E},\psi _E) = \omega _{L/E}(e) = \omega _{K/F}(-e^2),$ since $\omega _{L/E}$ is a conjugate-orthogonal character of $E^{\times }$.

For $i< j$, a similar computation using the above facts shows that

\begin{align*} \epsilon(1/2, {\rm Ind}_L^E (M_i^{\tau} \otimes M_j), \psi_E) =1 \end{align*}

Hence, we have

\begin{align*} \epsilon(1/2, {\rm As}(M) \cdot\mu^{-1}, \psi_E) &= \prod_i \chi_i(-1) \cdot \omega_{E/F}(-1)^d \cdot \omega_{K/F}(-e^2)^d \\ &\quad \cdot \prod_i \epsilon(1/2, \chi_i/ \chi_i^{\tau \sigma} \cdot (\mu \circ N_{L/E})^{-1}, \psi_L). \end{align*}

On the other hand, using Lemma 8.2(d),

\begin{align*} \det({\rm As}(M))(e) \cdot \omega_{K/F}(e^2)^{n(n-1)/2} = \omega_{K/F}(-1)^d. \end{align*}

Hence, Conjecture 8.3(iii) predicts that

\begin{align*} \mu(\det(V)) = \prod_i \chi_i(-1) \cdot \omega_{E/F}(-1)^d \cdot \omega_{K/F}(e^2)^d \cdot \prod_i \epsilon(1/2, \chi_i/ \chi_i^{\tau \sigma} \cdot (\mu \circ N_{L/E})^{-1}, \psi_L). \end{align*}

Comparing this with (10.10) and noting that

\begin{align*} \omega_{E/F}(k^2) = (e^2, k^2) = \omega_{K/F}(e^2), \end{align*}

we see that Conjecture 8.3(iii) holds for the $L$-packet defined by unitary principal series representations of ${\rm U}(V_K)$.

The reader will notice that we have not shown Conjecture 8.3(iv). For this, one would need to explicate which irreducible summand of the unitary principal series representation $\Pi$ has non-zero contribution to ${\rm Hom}_{{\rm U}(V)}(\Pi, \omega _{V, \mu, \psi })$. The different summands of $\Pi$ can be distinguished from each other by the effects on the normalized standard intertwining operators (i.e. the so-called local intertwining relations). We do not know how to exploit this to establish Conjecture 8.3(iv). However, in a paper [CG22] of Rui Chen and the first author, this remaining issue is taken care of by means of theta correspondence.

11. When $E \ne K$; global case

In this final section, we will formulate the global conjecture in the general case where $E \ne K$ are two distinct quadratic extensions of a global field $F$. We will use the notation of § 2.2 in this global setting.

Let $\Pi$ be a cuspidal automorphic representation of ${\rm U}(V_K)$ with a generic global $L$-parameter $M_{\Pi }$, so that

\begin{align*} M_{\Pi} = \bigoplus_{i=1}^d M_i, \end{align*}

is a sum of conjugate-dual cuspidal representations $M_i$ of ${\rm GL}_{m_i}(L \otimes \mathbb {A}_F)$ of sign $(-1)^{n-1}$ where $L= E \otimes _F K$. Now we have the global period integral

\begin{align*} \mathcal{P}: \Pi \otimes \omega_{V,\psi,\mu} \longrightarrow \mathbb{C} \end{align*}

defined as in § 2.2. The global conjecture is as follows.

Conjecture 11.1 The global period integral $\mathcal {P}$ is non-zero if and only if

1. (a) for all places $v$ of $F$, ${\rm Hom}_{{\rm U}(V_v)}(\Pi _v, \omega _{V_v,\psi, \mu _v}) \ne 0$;

2. (b) the twisted Asai automorphic $L$-function [Fli88] satisfies

   \begin{align*} L(1/2, \Pi, {\rm As}_{L/E} \times \mu^{-1}) \ne 0. \end{align*}
   (denoting $V_v=V\otimes _F F_v$).

Further, if $L(1/2, \Pi, {\rm As}_{L/E} \times \mu ^{-1}) \ne 0$, then there exists a skew-Hermitian space $V$ of dimension $n$ over $E$ such that the global period integral $\mathcal {P}$ is non-zero.

As in § 2.3, after fixing decompositions of Tamagawa measures and Petersson inner products, one expects a refined conjecture of the following form:

\begin{align*} \mathcal{P} \otimes \overline{\mathcal{P}} = \frac{1}{|S_{\Pi}|} \cdot \frac{L(1, M^{\vee}_{{\rm U}(V_K)})}{L(1, M^{\vee}_{{\rm U}(V)})} \cdot \frac{L(1/2, \Pi, {\rm As}_{L/E} \times \mu^{-1})}{L(1, \Pi, Ad)} \cdot \prod_v \mathcal{I}_v^{\#}. \end{align*}

Here:

• • $\mathcal {I}_v^{\#}$ is a normalized local period integral

  \begin{align*} \mathcal{I}_v^{\#} = \frac {L(1, M^{\vee}_{{\rm U}(V_v)})}{L(1, M^{\vee}_{{\rm U}(V_{K,v})})} \cdot \frac {L(1, \Pi_v, Ad)}{L(1/2, \Pi_v, {\rm As}_{L_v/E_v} \times \mu_v^{-1})} \cdot \mathcal{I}_v \end{align*}
  with
  \begin{align*} \mathcal{I}_v : \Pi_v \otimes \overline{\Pi_v} \otimes \overline{\omega_{\psi_v, \mu_v, V_v} } \otimes \omega_{\psi_v, \mu_v, V_v} \longrightarrow \mathbb{C} \end{align*}
  defined by the integral of matrix coefficients as in (2.3);

• • $M_{{\rm U}(V)}^{\vee }$ and $M_{{\rm U}(V_K)}^{\vee }$ are the dual of the motives of ${\rm U}(V)$ and ${\rm U}(V_K)$, respectively;

• • $|S_{\Pi }| = 2^d$, with $M_{\Pi } = \bigoplus _{i=1}^d M_i$.

Observe that for this global conjecture, all the local possibilities for $(E_v, K_v)$ will occur. It is conceivable that one can develop a relative trace formula, as in the case of GGP, to address the global conjectures here. This is being pursued by Danielle Wang, a PhD student of Wei Zhang at MIT.

Just as for Conjecture 2.7, we can verify the refined conjecture above when $\dim _E V =1$. More precisely, suppose one starts with a Hecke character $\chi$ of $\mathbb {A}_{L^{\times }}$, so that $\chi |_{\mathbb {A}_L^1}$ is an automorphic character of ${\rm U}(V_K)$. Then, as in the verification of Conjecture 2.7 in § 3.3, the period $\mathcal {P}$ is the (conjugate of) a global theta lift of $\chi |_{\mathbb {A}_E^1}$ from ${\rm U}(V)$ to ${\rm U}(W)$, where $W = \langle 1\rangle$ is a rank-one Hermitian space for $E/F$. Hence, for $\phi _1, \phi _2 \in \omega _{V,\psi,\mu }$, one has

\begin{align*} \mathcal{P}(\phi_1) \cdot \overline{\mathcal{P}(\phi_2)} \cdot \tau({\rm U}(W)) = \langle \Theta(\phi_2, \chi|_{E_1}) , \Theta(\phi_1, \chi|_{E_1}) \rangle_{{\rm U}(W), {\rm Pet}}. \end{align*}

The right-hand side is computed by the Rallis inner product formula, which gives

(11.2)\begin{equation} \mathcal{P}(\phi_1) \cdot \overline{\mathcal{P}(\phi_2)} = \frac{1}{2} \cdot \frac{L_E(1/2, (\chi^{\sigma} \chi^{-1})|_E \cdot \mu^{-1})}{L(1, \omega_{E/F})} \cdot \prod_v \mathcal{I}_v^{\#}(\chi,\chi, \phi_1,\phi_2), \end{equation}

taking note that the Tamagawa number $\tau ({\rm U}(W))$ is equal to $2$. This is precisely what the refined conjecture says in this case, since $|S_{\chi |_{L_1}}| =2$ here and

\begin{align*} {\rm As}_{L/E}(\chi^{\sigma} \chi^{-1}) =( \chi^{\sigma} \cdot \chi^{-1})|_E \end{align*}

by Lemma 8.2(d).

It is interesting to note that, as a sesquilinear form on $\omega _{V,\psi,\mu }$, the left-hand side of (11.2) is exactly the same as the left-hand side of (3.3) (assuming that the character $\chi$ restricts to the same character on $\mathbb {A}_E^1$ in the two cases). Moreover, the two ratio of $L$-values appearing on the right-hand side of (3.3) and (11.2) are exactly the same. Hence, the reader may wonder why there is a factor of $1/2$ on the right-hand side of (11.2) but there is none on the right-hand side of (3.3). The reason is that the adelic periods $\mathcal {I}^{\#} = \prod _v \mathcal {I}_v^{\#}$ on the right-hand side of both equations are different. Indeed, in view of (2.6), the adelic period $\mathcal {I}^{\#}$ in (3.3) is defined relative to the Petersson inner product $\langle -, - \rangle _{{\rm GL}(V), {\rm Pet}}$ of ${\rm GL}(V)$ whereas that in (11.2) is defined using the Petersson inner product $\langle -, -\rangle _{{\rm U}(V_K), {\rm Pet}}$ of ${\rm U}(V_K)$. As inner products on the one-dimensional vector space defined by $\chi$, the latter is twice the former, so that (3.3) and (11.2) are consistent with each other.