A formula for certain Shalika germs of ramified unitary groups

Abstract

In this article, for nilpotent orbits of ramified quasi-split unitary groups with two Jordan blocks, we give closed formulas for their Shalika germs at certain equi-valued elements with half-integral depth previously studied by Hales. Associated with these elements are hyperelliptic curves defined over the residue field, and the numbers we obtain can be expressed in terms of Frobenius eigenvalues on the first $\ell$-adic cohomology of the curves, generalizing previous result of Hales on stable subregular Shalika germs. These Shalika germ formulas imply new results on stability and endoscopic transfer of nilpotent orbital integrals of ramified unitary groups. We also describe how the same numbers appear in the local character expansions of specific supercuspidal representations and consequently dimensions of degenerate Whittaker models.

1 Introduction

We begin by introducing the unitary groups, related algebraic groups, Lie algebras and representations, and then the notion of Shalika germs. After that we can state our Shalika germ formulas, and describe their applications.

Let $F$ be a non-archimedean local field and $k$ its residue field. We fix an algebraic closure $\bar{k}$ of $k$ . We assume $\text{char}(k)\not =2$ . Let $E$ be a ramified quadratic extension over $F$ . Note $E/F$ is tame. Fix in this article a uniformizer $\unicode[STIX]{x1D70B}\in F$ whose square root $\unicode[STIX]{x1D70B}^{1/2}\in E$ . Let $n\geqslant 1$ be an integer and let $\tilde{G}=U_{n}(E/F)$ be the quasi-split unitary group of $n$ variables over $F$ which splits over  $E$ . We also assume either $\text{char}(F)=0$ or $\text{char}(F)>n$ .

The reason for the notation $\tilde{G}$ is that we prefer to, just like Bruhat–Tits and in geometric Langlands, think of reductive groups over $F$ intuitively as an ind-pro-scheme over the residue field $k$ . For this reason, in this article everything (groups, Lie algebras and their elements) that lives over $F$ will have its notation with a tilde $\widetilde{\;\;\;}$ .

Fix a vertex $\mathbf{x}$ on the Bruhat–Tits building of $\tilde{G}$ over $F$ whose reductive quotient is $\text{SO}_{n}(k)$ . The vertex $\mathbf{x}$ becomes hyperspecial after base change to $E$ . The reductive quotient at $\mathbf{x}$ over $E$ is (the $k$ -points of) $G:=\text{GL}_{n}$ . The root system of $G$ is in canonical bijection with the root system of $\tilde{G}/_{E}$ , and we can choose compatible pinnings for $G$ and $\tilde{G}/_{E}$ . The non-trivial element in $\text{Gal}(E/F)$ then provides an involution $\unicode[STIX]{x1D703}$ on $G$ such that the reductive quotient at $\mathbf{x}$ over $F$ is $(G^{\unicode[STIX]{x1D703}})^{o}(k)\cong \text{SO}_{n}(k)$ . A detailed and general construction of this is described in [RY14, § 4].

Write $\tilde{\mathfrak{g}}=\text{Lie }\tilde{G}$ and $\mathfrak{g}=\text{Lie }G$ . The involution $\unicode[STIX]{x1D703}$ also acts on $\mathfrak{g}$ . We will write $G(0)=(G^{\unicode[STIX]{x1D703}})^{o}\cong \text{SO}_{n}/_{k}$ , $\mathfrak{g}(0)=\mathfrak{g}^{\unicode[STIX]{x1D703}=1}$ , and $\mathfrak{g}(1)=\mathfrak{g}^{\unicode[STIX]{x1D703}=-1}$ . This can be realized as a $\mathbb{Z}/2$ -grading on  $\mathfrak{g}$ , and notationally we put $\mathfrak{g}(d):=\mathfrak{g}(d\text{ mod }2)$ for $d\in \mathbb{Z}$ . Write $V$ for the $n$ -dimensional standard representation of $G(0)$ and $\mathfrak{g}(0)$ . We have $\mathfrak{g}(1)\cong \operatorname{Sym}^{2}(V)$ as $G(0)$ -representations. The Moy–Prasad filtration associated to $\mathbf{x}$ jumps at half-integral numbers, and satisfies $\tilde{G}(F)_{\mathbf{x},0}/\tilde{G}(F)_{\mathbf{x},1/2}\cong G(0)(k)$ , $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},(d+1)/2}\cong \mathfrak{g}(d)(k)$ , for all $d\in \mathbb{Z}$ , where the latter isomorphism is as $G(0)(k)$ -representations.

Write $\mathfrak{g}(1)^{\text{rs}}:=\mathfrak{g}^{\text{rs}}\cap \mathfrak{g}(1)$ where $\mathfrak{g}^{\text{rs}}$ is the open subset of regular semisimple elements in the Lie algebra. Fix from now on a $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ . We can see $T$ as a self-adjoint endomorphism on $V$ . The monic characteristic polynomial $p_{T}$ is a separable polynomial of degree $n$ . Consequently $C_{T}:=(y^{2}=p_{T}(x))$ is a hyperelliptic curve with genus $g=\lfloor (n-1)/2\rfloor$ . In fact, the representation $G(0)\curvearrowright \mathfrak{g}(1)$ was first considered by Bhargava and Gross [BG14] for the study of arithmetic statistics about these hyperelliptic curves.

Consider the quotient map $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}{\twoheadrightarrow}\mathfrak{g}(1)(k)$ . Let $\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ be any lift. Such a $\tilde{T}$ is always regular semisimple and elliptic, i.e.  $\text{Stab}_{\tilde{G}}(\tilde{T})$ is an anisotropic torus over $F$ . In fact, it is even anisotropic over $F^{ur}$ . This implies that the orbits in the stable orbit of $\tilde{T}$ enjoy a bijection with the orbits in the stable orbit of $T$ ; see Lemma 4.5. (The notion of stable orbit will be reviewed in § 2.)

Denote by ${\mathcal{O}}(0)$ the set of nilpotent orbits in $\tilde{\mathfrak{g}}(F)$ , and $J(\tilde{X},f)$ the orbital integral of $f$ on the orbit of $\tilde{X}\in \tilde{\mathfrak{g}}$ . We will often identify an element in $\tilde{\mathfrak{g}}(F)$ with its orbit when talking about orbital integrals and Shalika germs. The theorem of Shalika [Sha72] asserts, for $\text{char}(F)=0$ or $\text{char}(F)\gg 0$ , the existence of constants, the Shalika germs $\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T})\in \mathbb{Q}$ such that

(1.1) $$\begin{eqnarray}J(\tilde{T},f)=\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0)}\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T})J({\mathcal{O}},f),\end{eqnarray}$$

for any compactly supported functions $f$ on $\tilde{\mathfrak{g}}(F)$ that are locally constant by a sufficiently large lattice.

In this article, we prove the following theorem for Shalika germs of nilpotent elements ${\tilde{N}}_{m}\in \tilde{\mathfrak{g}}(F)$ , $0\leqslant m\leqslant g$ with two Jordan blocks of sizes $n-m$ and $m$ . Denote by $q:=\#k$ . Let $\unicode[STIX]{x1D706}_{1},\unicode[STIX]{x1D706}_{1}^{\prime },\ldots ,\unicode[STIX]{x1D706}_{g},\unicode[STIX]{x1D706}_{g}^{\prime }$ be Frobenius eigenvalues on $H^{1}(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell })$ , ordered so that $\unicode[STIX]{x1D706}_{i}\unicode[STIX]{x1D706}_{i}^{\prime }=q$ . Also write $\unicode[STIX]{x1D706}_{0}=1$ , $\unicode[STIX]{x1D706}_{0}^{\prime }=q$ . Let $I=\{1,\ldots ,g\}$ if $n=2g+1$ and $I=\{0,1,\ldots ,g\}$ if $n=2g+2$ . Write

$$\begin{eqnarray}a_{m}(T):=(-1)^{m}\cdot \!\!\mathop{\sum }_{S\subset I,|S|=m}\bigg(\mathop{\prod }_{i\in S}(\unicode[STIX]{x1D706}_{i}+\unicode[STIX]{x1D706}_{i}^{\prime })\bigg).\end{eqnarray}$$

Theorem 1.1 (Theorems 4.1 and 4.11).

For $0\leqslant m\leqslant g$ , we have $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}^{\text{st}}(\tilde{T})=\pm a_{m}(T)$ for any lift $\tilde{T}$ of $T$ , where $\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}^{\text{st}}(\tilde{T})$ is the sum of  $\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T}^{\prime })$ for $\tilde{T}^{\prime }$ running over different orbits in the stable orbit of $\tilde{T}$ .

See the theorems for the sign and see Appendix A for the normalization. When $m=0$ , ${\tilde{N}}_{0}$ is a regular orbit and $a_{0}(T)=1$ which is a well-known result of Shelstad [She89]. When $m=1$ it is a subregular orbit, and the result was proven by Hales [Hal94]. He also gave parallel results for other classical groups. Our result probably brings the suggestion that general Shalika germs, after all, could have reasonably nice closed formulas.

The starting point of the proof of Theorem 1.1 is to find a particular sequence of test functions for (1.1) supported on $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ . These functions are made available by the homogeneity result of DeBacker (special case by Waldspurger) [DeB02]. The description of these functions will be given in the beginning of § 4. For these test functions, the left-hand side of (1.1) counts $k$ -points on a sequence of specific varieties. It turns out that the theory of pencils of quadrics by Wang [Wan13] can be used to relate these varieties to $\operatorname{Sym}^{m}(C_{T})$ , the $m$ th symmetric power of the hyperelliptic curve $C_{T}$ . We are thus brought to the numbers $a_{m}(T)$ above. This is the main concept in § 3.

Already in Goresky et al. [GKM06] we knew certain orbital integrals can be understood via counting points on Hessenberg varieties (their definition of Hessenberg varieties are more general than others’, see [GKM06, 1.5]). The situation for obtaining Shalika germs could be slightly more involved and our method ends up with counting points on certain étale schemes over Hessenberg varieties (there are some essential difference between our methods, though). A template of such is given in [Tsa15b, § 4]. It then happens that our étale schemes over Hessenberg varieties are strongly related to varieties considered by Wang.

Section 4 contains most of the computation. We begin with the case of odd ramified unitary groups. In § 4.1 we read out the varieties that appear in the left-hand side of (1.1) for our test functions and apply the geometric result in § 3. Next in § 4.2 we use Ranga Rao’s method to compute nilpotent orbital integrals. With our simple-looking test functions thanks to homogeneity result of DeBacker, our computation reduces to a combinatorial sum over the Weyl group of $G(0)$ . In § 4.3 we state results regarding Shalika germs (instead of stable Shalika germs), as well as the results for even quasi-split ramified unitary groups.

A consequence of Theorem 1.1 is a result regarding stable distributions supported on the nilpotent cone (i.e. linear combination of nilpotent orbital integrals) and endoscopic transfer of nilpotent orbital integrals of ramified quasi-split unitary groups. This is the main content of § 5. The basic idea is that Shalika germs are the coefficients comparing regular semisimple orbital integrals and nilpotent orbital integrals. Once we know these coefficients, we are able to derive, from the very definition of stability and endoscopic transfer of regular semisimple orbital integrals, corresponding results of nilpotent ones.

The relevant elliptic endoscopic data are $U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)$ with $n_{1}+n_{2}=n$ as endoscopy groups of $U_{n}(E/F)$ . Assuming some conjectures of Assem (Conjectures 5.1 and 5.5), our result for nilpotent orbits with two Jordan blocks agrees with previous results of Waldspurger [Wal01] for unramified unitary groups. This also provides another evidence for Assem’s conjectures. In fact, it was this connection to endoscopic transfer which led us into believing the formula in Theorem 1.1 in the first place (see Remark 5.11).

In addition, in § 6 we describe how those Shalika germs we compute show up in the Harish-Chandra–Howe local character expansions for some supercuspidal representations. Since Mœglin and Waldspurger [MW87] showed that the coefficients in the local character expansions are related to the dimension of certain degenerate Whittaker models, we can produce examples where the dimension of degenerate Whittaker model are given by counting points on some ‘non-elementary’ varieties.

2 Notation and setup

We collect the notation. We have a non-archimedean local field $F$ , its residue field $k$ , a ramified quadratic extension $E/F$ , and a fixed uniformizer $\unicode[STIX]{x1D70B}\in F$ such that $\unicode[STIX]{x1D70B}^{1/2}\in E$ . Also we denote $q:=\#k$ . We have $\tilde{G}=U_{n}(E/F)$ is a quasi-split unitary group that splits over $E$ (such group is unique). We write $\tilde{\mathfrak{g}}=\text{Lie }\tilde{G}$ . When orbital integrals on $\tilde{G}(F)$ or $\tilde{\mathfrak{g}}(F)$ are concerned, we always identify an element with its $\tilde{G}(F)$ -orbit. The assumptions $\text{char}(k)\not =2$ and either $\text{char}(F)=0$ or $\text{char}(F)>n$ are imposed. In fact, we will mostly work with the assumption $\text{char}(k)\gg 0$ , and leave it to Appendix C to explain how we can reduce the assumption on characteristic to those stated above.

Write $G=\text{GL}_{n}/_{k}$ . It has a standard representation $V$ , which we equipped with a non-degenerate quadratic form $\langle \cdot \,,\cdot \rangle$ . We define an involution $\unicode[STIX]{x1D703}$ on $G$ such that $\unicode[STIX]{x1D703}(h)=(h^{\text{t}})^{-1}$ for $h\in G$ , where $h^{\text{t}}$ is the transpose of $h$ with respect to $\langle \cdot \,,\cdot \rangle$ . This induces an involution on $\mathfrak{g}:=\text{Lie }G$ which we also denote by $\unicode[STIX]{x1D703}$ . Let $G(0)=(G^{\unicode[STIX]{x1D703}})^{o}\cong \text{SO}_{n}$ , $\mathfrak{g}(0)=\mathfrak{g}^{\unicode[STIX]{x1D703}=1}=\text{Lie }G(0)$ and $\mathfrak{g}(1)=\mathfrak{g}^{\unicode[STIX]{x1D703}=-1}$ the invariant and anti-invariant subspace of $\unicode[STIX]{x1D703}$ . We write $\mathfrak{g}(d):=\mathfrak{g}(d\text{ mod }2)$ for $d\in \mathbb{Z}$ . This gives a $\mathbb{Z}/2$ -grading on $\mathfrak{g}$ , that is, $[\mathfrak{g}(d_{1}),\mathfrak{g}(d_{2})]\subset \mathfrak{g}(d_{1}+d_{2})$ . There is a vertex $\boldsymbol{x}$ on the building such that $G$ is the reductive quotient of $\tilde{G}/_{E}$ at $x$ and $G(0)$ the reductive quotient of $\tilde{G}/_{F}$ . We fix such a vertex $\boldsymbol{x}$ . Also see § 2.1 below for a more elementary description of $\tilde{G}$ , $G$ and $\mathbf{x}$ .

Let $\mathfrak{g}^{\text{rs}}\subset \mathfrak{g}$ be the subset of regular semisimple elements and $\mathfrak{g}(1)^{\text{rs}}=\mathfrak{g}^{\text{rs}}\cap \mathfrak{g}(1)$ . For any $T\in \mathfrak{g}(1)^{\text{rs}}$ , the monic characteristic polynomial of $T$ is denoted $p_{T}(x)$ , and $C_{T}=(y^{2}=p_{T}(x))$ is the smooth completion of the hyperelliptic curve defined by $p_{T}(x)$ .

Whenever we have a group variety $H$ acting on a space $X$ over some field $K$ , by an orbit (or the orbit of $x\in X(K)$ ) in $X(K)$ we mean a subset of $X(K)$ of the form $\{h.x\mid h\in G(K)\}$ , and by a stable orbit (or the stable orbit of $x$ ) we mean a subset of $X(K)$ of the form $\{h.x\mid h\in G(K^{\text{sep}})\}\cap X(K)$ . The (stable) orbits discussed in this article will be either (stable) orbits in $\tilde{\mathfrak{g}}(F)$ under the adjoint action of $\tilde{G}$ , or (stable) orbits in $\mathfrak{g}(1)(k)$ under the conjugacy action of $G(0)$ .

The methods for odd ramified unitary groups ( $n=2g+1$ ) and even (quasi-split) ramified unitary group ( $n=2g+2$ ) are largely the same, but most of the computation has to be carried out separately. In most of this article we only treat the odd case in detail, but describe geometric tools needed for even unitary groups and list the results. In particular, we will go with $\tilde{G}=U_{2g+1}(E/F)$ unless otherwise stated, and notationally reserve $n$ for other variables.

2.1 An elementary description

We give a down-to-earth description of groups $\tilde{G}$ , $G$ , the involution $\unicode[STIX]{x1D703}$ and the vertex $\mathbf{x}$ . Let $\tilde{V}$ be an $n$ -dimensional hermitian space over $E$ , spanned by basis vectors $\tilde{e}_{1},\ldots ,\tilde{e}_{n}$ and equipped with the hermitian form given by $\langle \sum a_{i}\tilde{e}_{i},\sum b_{i}\tilde{e}_{i}\rangle _{\text{herm}}=\sum _{i=1}^{n}a_{n+1-i}b_{i}^{\ast }$ , where $a_{i},b_{i}\in E$ and $b_{i}^{\ast }$ is the conjugate of $b_{i}$ over $F$ . Then $\tilde{G}$ is such an algebraic group defined over $F$ for which $\tilde{G}(F)$ is isomorphic to the group of unitary operators on $\tilde{V}$ , i.e.  $E$ -linear operators on $\tilde{V}$ preserving the hermitian form.

Let $\unicode[STIX]{x1D6EC}=\text{span}_{{\mathcal{O}}_{E}}\{\tilde{e}_{1},\ldots ,\tilde{e}_{n}\}$ be a lattice in $\tilde{V}$ . Let $K$ be the subgroup of $\tilde{G}(F)$ consisting of unitary operators $g$ with $g(\unicode[STIX]{x1D6EC})=\unicode[STIX]{x1D6EC}$ . Then $K$ stabilizes a unique vertex on the Bruhat–Tits building of $\tilde{G}$ over $F$ , which (up to conjugation) is the vertex that we call $\mathbf{x}$ . We have the stabilizer group $\tilde{G}(F)_{\mathbf{x}}=K$ .

The hermitian form $\langle \cdot \,,\cdot \rangle _{\text{herm}}$ takes ${\mathcal{O}}_{E}$ values on $\unicode[STIX]{x1D6EC}$ . Its reduction mod $\unicode[STIX]{x1D70B}^{1/2}$ thus defines a quadratic form $\langle \cdot \,,\cdot \rangle$ on $V:=\unicode[STIX]{x1D6EC}/\unicode[STIX]{x1D70B}^{1/2}\unicode[STIX]{x1D6EC}$ . Write $e_{1},\ldots ,e_{n}$ to be the reduction of $\tilde{e}_{1},\ldots ,\tilde{e}_{n}$ , respectively. Then $\langle \cdot \,,\cdot \rangle$ on $V$ is defined by $\langle \sum a_{i}e_{i},\sum b_{i}e_{i}\rangle =\sum _{i=1}^{n}a_{n+1-i}b_{i}$ , where $a_{i},b_{i}\in k$ . The algebraic group $G$ then should be identified with the group of automorphisms of $V$ (not necessarily fixing $\langle \cdot \,,\cdot \rangle$ ); $G(k^{\prime })=\text{GL}(V\otimes _{k}k^{\prime })$ for any finite extension $k^{\prime }/k$ , and $\unicode[STIX]{x1D703}\curvearrowright G(k^{\prime })$ is the involution $\unicode[STIX]{x1D703}(g)=(g^{\text{t}})^{-1}$ where $g^{\text{t}}$ denotes the transpose of $g$ with respect to the quadratic form $\langle \cdot \,,\cdot \rangle$ . The differential of $\unicode[STIX]{x1D703}$ , still denoted by $\unicode[STIX]{x1D703}$ acts on $\mathfrak{g}=\text{Lie }G$ by $\unicode[STIX]{x1D703}(X)=-X^{\text{t}}$ .

The Lie algebra $\tilde{\mathfrak{g}}(F)$ is the space of anti-hermitian endomorphisms of $\tilde{V}$ . For any $d\in \frac{1}{2}\mathbb{Z}$ , one has the attached Moy–Prasad sublattice $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d}=\{X\in \tilde{\mathfrak{g}}(F)\mid X(\unicode[STIX]{x1D6EC})\subset \unicode[STIX]{x1D70B}^{d}\unicode[STIX]{x1D6EC}\}$ .

Lemma 2.1. For any $d\in \frac{1}{2}\mathbb{Z}$ , we have a natural isomorphism $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},d+1/2}\cong \mathfrak{g}(2d)(k)$ .

Proof. Let $r:\operatorname{End}_{{\mathcal{O}}_{E}}(\unicode[STIX]{x1D6EC})\rightarrow \operatorname{End}_{k}(V)$ be the natural reduction map. The map $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d}\rightarrow \mathfrak{g}(k)$ is given by $X\mapsto r(\unicode[STIX]{x1D70B}^{-d}X)$ . When $d\in \mathbb{Z}$ , $\unicode[STIX]{x1D70B}^{-d}X$ is anti-hermitian and, thus, $r(\unicode[STIX]{x1D70B}^{-d}X)$ is anti-self-adjoint, i.e.  $r(\unicode[STIX]{x1D70B}^{-d}X)\subset \mathfrak{g}(2d)=\mathfrak{g}(0)$ . When $d\in \frac{1}{2}+\mathbb{Z}$ , $\unicode[STIX]{x1D70B}^{-d}X$ is hermitian and therefore $r(\unicode[STIX]{x1D70B}^{-d}X)\subset \mathfrak{g}(2d)=\mathfrak{g}(1)$ is self-adjoint. It is easy to see that the resulting map $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d}\rightarrow \mathfrak{g}(2d)(k)$ is surjective and trivial on $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d+1/2}$ , giving the asserted isomorphism.◻

Note that the isomorphism above depends on the choice of uniformizer $\unicode[STIX]{x1D70B}^{1/2}\in E$ . Lastly, the algebraic group $G$ has $\unicode[STIX]{x1D703}$ -stable Borel subgroups defined over $k$ . One such $B$ is given by that $B(k)$ consists of endomorphisms of $V$ that sends $e_{i}$ to a linear combination of $e_{1}$ , $e_{2},\ldots ,$ and $e_{i}$ . We also denote $B(0)=B\cap G(0)=(B^{\unicode[STIX]{x1D703}})^{o}$ . They are used in § 4. For $n\geqslant 3$ , all such $\unicode[STIX]{x1D703}$ -stable Borel subgroups are $G(0)$ -conjugate.

3 Geometric result via pencils of quadrics

In this section, $k$ can be any perfect field with $\text{char}(k)\not =2$ .

3.1 Odd case

In this subsection we have $n=2g+1$ and $G=\text{GL}_{2g+1}/_{k}=\text{GL}(V)$ . Recall that the vector space $V$ comes with a non-degenerate quadratic form $\langle \cdot \,,\cdot \rangle$ . We then have in the introduction an involution $\unicode[STIX]{x1D703}$ on $G$ which sends $g$ to $(g^{\text{t}})^{-1}$ , where $g^{\text{t}}$ is the adjoint of $g$ with respect to $\langle \cdot \,,\cdot \rangle$ . This induces an involution on $\mathfrak{g}$ , and we write $\mathfrak{g}(0)=\mathfrak{g}^{\unicode[STIX]{x1D703}=1}$ , $\mathfrak{g}(1)=\mathfrak{g}^{\unicode[STIX]{x1D703}=-1}$ . We have $\mathfrak{g}(1)\cong \operatorname{Sym}^{2}(V)$ as $G(0)$ -representations. As $\langle \cdot \,,\cdot \rangle$ provides a self-dual structure on $V$ , $\mathfrak{g}(1)\cong \text{End}^{\text{self}-\text{adj}}(V)$ is also the space of self-adjoint operators on $V$ .

The representation $G(0)\curvearrowright \mathfrak{g}(1)$ , or equivalently $\text{SO}(V)\curvearrowright \operatorname{Sym}^{2}(V)$ , was considered by Bhargava and Gross in [BG14]. An orbit in this representation is GIT-stable if and only if it is contained in $\mathfrak{g}(1)^{\text{rs}}:=\mathfrak{g}^{\text{rs}}\cap \mathfrak{g}(1)$ where $\mathfrak{g}^{\text{rs}}$ is the open subset of regular semisimple elements in the Lie algebra. We now fix an $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ .

Let $p_{T}(x)$ be the degree $2g+1$ monic characteristic polynomial of $T$ . Let $L=k[x]/p_{T}(x)$ be a degree $2g+1$ étale algebra over $k$ . Consider the Weil restriction $\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ . This is a commutative étale finite group scheme over $k$ of order $2^{2g+1}$ . It has a surjective norm map $Nm:\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\rightarrow \unicode[STIX]{x1D707}_{2}$ . Bhargava and Gross observed for $T\in \mathfrak{g}(1)^{\text{rs}}$ , we have canonical isomorphism $\text{Stab}_{G(0)}(T)\cong \text{ker}(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . In fact, the map $T\mapsto p_{T}(x)$ is the GIT-quotient map $\mathfrak{g}(1)\mapsto \mathfrak{g}(1)/\!/G(0)$ ; we have $\mathfrak{g}(1)/\!/G(0)\cong \mathbb{A}^{2g+1}$ is the space of degree $n$ monic polynomials.

Let $C_{T}=(y^{2}=p_{T}(x))$ be a (smooth completion of) genus $g$ hyperelliptic curve. Let $J_{T}=\operatorname{Pic}^{0}(C_{T})$ . Since the $2$ -torsion $J_{T}[2]$ is generated by differences of Weierstrass points, one checks $J_{T}[2]\cong \text{ker}(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . Consequently $J_{T}[2]\cong \text{Stab}_{G(0)}(T)$ .

If one fix such a $T$ , then the orbit of $T$ is $G(0)(k).T$ while the stable orbit of $T$ is $(G(0)(\bar{k}).T)\cap \mathfrak{g}(1)(\bar{k})$ . There could be more than one orbits inside a stable orbit, and relative to the choice of $T$ as a pinning they can be classified by $\ker (H^{1}(k,\text{Stab}_{G(0)}(T))\rightarrow H^{1}(k,G(0)))$ . When $k$ is a finite field, by Lang’s theorem, the latter pointed set is trivial, and thus we have $H^{1}(k,\text{Stab}_{G(0)}(T))\cong H^{1}(k,J_{T}[2])$ classifies orbits in the stable orbit of $T$ relative to the choice of a pinning.

The GIT-quotient map $\mathfrak{g}(1)\rightarrow \mathfrak{g}(1)/\!/G(0)$ has a Kostant section [Lev09, Theorem 5.5]. Using the Kostant section as a pinning, a $G(0)(k)$ -orbit in $\mathfrak{g}(1)^{\text{rs}}(k)$ corresponds to a hyperelliptic curve $C_{T}$ together with a class in $H^{1}(k,J_{T}[2])$ . For $k$ a global field, Bhargava, Gross and others used this to study the average size of $2$ -Selmer groups of such hyperelliptic curves, see e.g. [BG13]. For this purpose, Wang developed the theory of pencil of quadrics [Wan13]. It turns out that his theory is very useful in describing the variety that we will encounter in orbital integrals.

On the vector space $V\oplus k$ we define two quadratic forms by $\langle (v_{1},c_{1}),(v_{2},c_{2})\rangle _{1}=\langle v_{1},v_{2}\rangle$ and $\langle (v_{1},c_{1}),(v_{2},c_{2})\rangle _{2}=\langle v_{1},Tv_{2}\rangle -c_{1}c_{2}$ . This defines a generic pencil of quadrics in the sense of Wang [Wan13, Introduction]. Recall that a subspace $W\subset V\oplus k$ is said to be isotropic with respect to a quadric (e.g.  $\langle \cdot \,,\cdot \rangle _{1}$ ) if the restriction of the quadratic form to $W$ is trivial. In his paper, Wang proved the following result.

Theorem 3.1 (Wang [Wan13, Theorem 2.26]).

Let $F_{T}$ be the variety that parameterizes $g$ -dimensional subspaces of $V\oplus k$ that the are isotropic with respect to both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ . Then there is a commutative algebraic group structure on

$$\begin{eqnarray}G_{T}:=J_{T}\sqcup F_{T}\sqcup \operatorname{Pic}^{1}(C_{T})\sqcup F_{T}^{\prime },\end{eqnarray}$$

where $F_{T}^{\prime }$ is a copy of $F_{T}$ as an abstract variety, and $G_{T}$ satisfies the following.

1. (i) The component group of $G_{T}$ is equal to $\mathbb{Z}/4$ . The four components above correspond to $0+4\mathbb{Z}$ , $1+4\mathbb{Z}$ , $2+4\mathbb{Z}$ and $3+4\mathbb{Z}$ , respectively.

2. (ii) The addition law on $J_{T}\sqcup \operatorname{Pic}^{1}(C_{T})$ agrees with the natural one on $\operatorname{Pic}(C_{T})/(2(\infty )=0)$ .

3. (iii) The inversion map of $G_{T}$ restricts to an isomorphism $F_{T}\xrightarrow[]{{\sim}}F_{T}^{\prime }$ .

In particular, $F_{T}$ is a torsor under $J_{T}$ and there is a doubling map $\times 2:F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ . We review the group structure in the theorem. The group structure is determined by $(p)-[W]$ , i.e. how to subtract from $p\in C_{T}$ a subspace $[W]\in F_{T}$ . This is done as follows: recall that a ruling is a connected component of the variety parameterizing $(g+1)$ -dimensional subspace on which the quadratic form is trivial. A point $p=(x,y)$ on $C_{T}$ corresponds to a ruling of $\langle \cdot \,,\cdot \rangle _{2}-x\langle \cdot \,,\cdot \rangle _{1}$ (see [Wan13, p. 8]). There will be a unique $(g+1)$ -dimensional space $W^{\prime }$ in the ruling such that $W^{\prime }\supset W$ . Inside the space $W^{\prime }$ there will be, when counted with multiplicity, two $g$ -dimensional subspaces $W$ and $W^{\prime \prime }$ on which both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish. It is then defined $(p)-[W]:=[W^{\prime \prime }]$ . This uniquely characterizes the group structure on $G_{T}$ .

We give an example of $(p)-[W]$ when $p=\infty$ , the rational point on $C_{T}$ at infinity. Note that $\infty \in C_{T}$ correspond to the quadric $\langle \cdot \,,\cdot \rangle _{1}$ , which is degenerate and has only one ruling. Let $\unicode[STIX]{x1D70F}_{\infty }:V\oplus k\rightarrow V\oplus k$ be the map that sends $(v,c)$ to $(v,-c)$ . It is obvious that if $[W]\in F_{T}$ , then $[\unicode[STIX]{x1D70F}_{\infty }W]\in F_{T}$ . Let us fix such $W$ and write $W^{\prime \prime }=\unicode[STIX]{x1D70F}_{\infty }W$ . As $W\subset V\oplus k$ is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{2}$ , the second component is not contained in $W$ . Hence, $W^{\prime }:=W+k$ , where this $k$ is the second component of $V\oplus k$ , is $(g+1)$ -dimensional, and is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{1}$ as $W$ is. This says that $W^{\prime }$ and $W^{\prime \prime }$ satisfy the properties in the previous paragraph, and consequently $(\infty )-[W]=[W^{\prime \prime }]=[\unicode[STIX]{x1D70F}_{\infty }(W)]$ .

It is obvious that $G_{T}$ depends only on $T$ up to $G(0)(k)$ -conjugacy. As mentioned in the introduction the orbit of $T$ in its stable orbit may be characterized by a class $\unicode[STIX]{x1D6FC}_{T}\in H^{1}(k,J_{T}[2])$ . This class can be described as follows: by Theorem 3.1(i) and (ii) the map $\times 2:F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ is étale and Galois with Galois group being $J_{T}[2]$ as a group scheme over $k$ . From the previous paragraph and [Wan13, Proposition 2.29], respectively, one has the following result.

Lemma 3.2. We have:

1. (i) $(\times 2)^{-1}(\infty )\subset F_{T}$ parameterizes $g$ -dimensional subspaces in $V$ that are isotropic with respect to $\langle \cdot \,,\cdot \rangle$ and $\langle \cdot ,T\cdot \rangle =\langle \cdot \,,\cdot \rangle _{2}|_{V}$ ;

2. (ii) the class $\unicode[STIX]{x1D6FC}_{T}$ is represented by the torsor $(\times 2)^{-1}(\infty )$ .

We will later need an explicit description of the $J_{T}[2]$ -action on $(\times 2)^{-1}(\infty )$ . Let $p_{0},\ldots ,p_{2g},\infty$ be the Weierstrass points of $C_{T}$ . Then $J_{T}[2]$ is generated by $((p_{i})-(\infty ))$ , $0\leqslant i\leqslant 2g$ with the only relation $\sum _{i=0}^{2g}((p_{i})-(\infty ))=0$ . The action of $(p_{i})-(\infty )$ can be described as follows. Say $p_{i}=(x,0)$ . Then $x\langle \cdot \,,\cdot \rangle _{1}-\langle \cdot \,,\cdot \rangle _{2}$ is a degenerate quadric with one-dimensional kernel $U$ . For $[W]\in (\times 2)^{-1}(\infty )$ , $x\langle \cdot \,,\cdot \rangle _{1}-\langle \cdot \,,\cdot \rangle _{2}$ is trivial on the $(g+1)$ -dimensional space $W+U$ . There is exactly one $g$ -dimensional subspace $W^{\prime }\subset W+U$ , other than $W$ , on which $\langle \cdot \,,\cdot \rangle$ and $\langle \cdot ,T\cdot \rangle$ are also trivial. The $J_{T}[2]$ -action on $(\times 2)^{-1}(\infty )$ is then given by the following result.

Lemma 3.3 [Wan13, Proposition 2.29].

With the notation above, $((p_{i})-(\infty )).[W]=[W^{\prime }]$ .

For any $0\leqslant m\leqslant g$ , consider $j_{m}:\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by $j_{m}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-(m-1)(\infty )$ . Let $X_{T,m}$ be the image of $j_{m}$ , and let $\tilde{X}_{T,m}:=(\times 2)^{-1}(X_{T,m})$ be its preimage under the étale map $\times 2$ . For example $\tilde{X}_{T,0}=(\times 2)^{-1}(\infty )$ . We also put $\tilde{X}_{T,-1}=\emptyset$ . We shall relate $\tilde{X}_{T,m}$ with the following varieties $F_{T,m}$ , which could be thought as a generalized version of Hessenberg varieties considered by Goresky et al. [GKM06].

For any finite extension $k^{\prime }/k$ , we call a flag of $k^{\prime }$ -subspaces $0=W^{0}\subset W^{1}\subset \cdots \subset W^{g}\subset (V\oplus k)\otimes _{k}k^{\prime }$ good if:

1. (i) $\dim W^{i}=i$ ;

2. (ii) the restriction of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ to $W^{g}$ is zero;

3. (iii) $W^{g-1}\subset V\otimes _{k}k^{\prime }$ ;

4. (iv) $T(W^{i})\subset W^{i+2}$ for all $1\leqslant i\leqslant g-3$ ;

5. (v) $T(W^{g-2})\subset \unicode[STIX]{x1D70B}_{1}(W^{g})$ .

Here $\unicode[STIX]{x1D70B}_{1}:(V\oplus k)\otimes _{k}k^{\prime }\rightarrow V\otimes _{k}k^{\prime }$ is the projection to the first factor. For $1\leqslant m\leqslant g$ , a good flag is called $m$ -good if $T(W^{g-m})\subset \unicode[STIX]{x1D70B}_{1}(W^{g-m+1})$ . Note this is simply $T(W^{g-m})\subset W^{g-m+1}$ when $m>1$ . Also a good flag is called $0$ -good if $W^{g}\subset V\otimes _{k}k^{\prime }$ . Next, for $0\leqslant m\leqslant g$ , an $m$ -good flag is called $m$ -excellent if it is also $n$ -good for $m<n\leqslant g$ . On the other hand, a good flag is called $m$ -general if it is not $n$ -good for any $0\leqslant n<m$ . Finally, a good flag is called $m$ -exact if it is $m$ -excellent and $m$ -general. Now let

$$\begin{eqnarray}F_{T,m}(k^{\prime })=\{0=W^{0}\subset W^{1}\subset \cdots \subset W^{g}\subset (V\oplus k)\otimes _{k}k^{\prime }\mid \text{This is an }m\text{-exact flag}\}.\end{eqnarray}$$

The functor $F_{T,m}$ is easily seen from its very definition to be represented by a quasi-projective variety over $k$ which we will denote with the same notation. In fact, there is a projective variety $F_{T,\text{good}}$ that parameterize good flags, and $F_{T,m}\subset F_{T,\text{good}}$ is locally closed. There is a natural map $\tilde{j}:F_{T,\text{good}}\rightarrow F_{T}$ by sending a flag to $[W^{g}]$ . In the rest of this section, we work ‘geometrically,’ i.e. we replace $k$ by an algebraic closure $\bar{k}$ , so that we can omit the notation $\cdot \otimes _{k}k^{\prime }$ and so on. This section is mostly devoted to the proof of the following result.

Theorem 3.4. For $0\leqslant m\leqslant g$ , the map $\tilde{j}|_{F_{T,m}}:F_{T,m}\rightarrow F_{T}$ is a locally closed embedding, with image equal to $\tilde{X}_{T,m}\,\backslash \,\tilde{X}_{T,m-1}$ .

Remark 3.5. A more direct proof of this theorem in the case $m\leqslant 2$ was shown to me by Wang. For demonstration purposes, we do the $m=1$ case, i.e. we prove $\text{im}(\tilde{j}|_{F_{T,1}})=\tilde{X}_{T,1}\,\backslash \,\tilde{X}_{T,0}$ . (This remark will not be used in the proof for the general case and can be safely skipped.) Recall that a $1$ -exact flag is $0=W^{0}\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ such that (i) $\dim W^{i}=i$ , (ii) $W^{g}$ is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ , (iii) $W^{g-1}\subset V$ , (iv) $T(W^{i})\subset W^{i+1}$ for $i=1,\ldots ,g-2$ , (v) $T(W^{g-1})\subset \unicode[STIX]{x1D70B}_{1}(W^{g})$ and (vi) $W^{g}\not \subset V$ (the first factor of $V\oplus k$ ). Note that condition (v) says the flag is $1$ -good and condition (vi) says that it is not $0$ -good. We will show that under these conditions, one has $2[W^{g}]=(p)\in \operatorname{Pic}^{1}(C_{T})\subset G_{T}$ for some $p\in C_{T}\backslash \{\infty \}$ . This establishes the above statement about the image, which is in fact the essential part; the rest of Theorem 3.4 for $m=1$ follows from Lemma 3.7.

Denote by $(W^{g})^{\bot 1}$ the orthogonal complement of $W^{g}$ in $V\oplus k$ with respect to $\langle \cdot \,,\cdot \rangle _{1}$ . By condition (ii) the second component $k$ of $V\oplus k$ cannot be contained in $W^{g}$ , and one deduces $\dim (W^{g})^{\bot 1}=\dim V\oplus k-\dim W^{g}=g+2$ . Since $\langle \cdot \,,\cdot \rangle _{2}|_{V}=\langle \cdot ,T\cdot \rangle$ , with condition (v) we observe $(W^{g-1})^{\bot 2}\subset (W^{g})^{\bot 1}$ . As $(W^{g})^{\bot 2}$ is contained in $(W^{g-1})^{\bot 2}$ with codimension at most $1$ , we have $(W^{g})^{\bot 1}\cap (W^{g})^{\bot 2}\subset (W^{g})^{\bot 1}$ with codimension at most $1$ . This is indeed a proper inclusion, as the second component $k\not \subset (W^{g})^{\bot 2}$ by condition (vi). In particular, $W^{\prime }:=(W^{g})^{\bot 1}\cap (W^{g})^{\bot 2}$ has dimension $g+1$ .

From condition (ii) we have $W^{\prime }\supset W^{g}$ . The restriction of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ on $W^{\prime }$ then factor through the one-dimensional quotient $W^{\prime }/W^{g}$ . In particular, some non-trivial linear combination of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ has to be trivial on $W^{\prime }/W^{g}$ and therefore trivial on $W^{\prime }$ . Put $W=W^{\prime \prime }:=W^{g}$ . The fact that the above linear combination, as a quadratic form, vanish to order two on $0\in W^{\prime }/W^{g}$ says that inside $W^{\prime }$ we have two $g$ -dimensional subspaces $W$ and $W^{\prime \prime }$ (counted with multiplicity) on which $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish. That is, $W$ , $W^{\prime }$ and $W^{\prime \prime }$ play the role given in the paragraph after Theorem 3.1. This means $(p)-[W]=[W^{\prime \prime }]=[W^{g}]$ for some $p\in C_{T}$ , i.e.  $2[W^{g}]=(p)$ . We also note that the above linear combination of quadratic forms cannot be a multiple of solely $\langle \cdot \,,\cdot \rangle _{1}$ because $k\not \subset (W^{g})^{\bot 2}\Rightarrow k\not \subset W^{\prime }\Rightarrow \dim (\unicode[STIX]{x1D70B}_{1}(W^{\prime }))=g+1$ . This implies $p\not =\infty$ and finishes the proof about $\text{im}(\tilde{j}|_{F_{T,1}})$ .

We now begin our proof for the general case with two simple lemmas.

Lemma 3.6. If $0\subsetneq W\subsetneq V$ is such that $\langle \cdot \,,\cdot \rangle$ is trivial on $W$ , then $T(W)\not =W$ .

Proof. Let $W^{\bot }=\{v\in V\mid \langle v,w\rangle =0,\;\forall w\in W\}$ . Then $W\subset W^{\bot }$ by assumption. Suppose on the contrary $T(W)=W$ , then by adjointness $T(W^{\bot })=W^{\bot }$ , and $T|_{W}$ is the adjoint of $T|_{V/W^{\bot }}$ . But this says $T|_{W}$ and $T|_{V/W^{\bot }}$ have the same eigenvalues. Hence, $T$ cannot be regular semisimple.◻

Lemma 3.7. Let $0\leqslant m\leqslant g$ .

1. (i) Suppose $0\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ and $0\subset (W^{1})^{\prime }\subset \cdots \subset (W^{g})^{\prime }\subset V\oplus k$ are such that $W^{g}=(W^{g})^{\prime }$ . If one of the flags is $m$ -general, then $W^{i}=(W^{i})^{\prime }$ for $g-m\leqslant i\leqslant g$ . In particular, the other is also $m$ -general.

2. (ii) If $0\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ is $m$ -good, then there is a unique $0\subset (W^{1})^{\prime }\subset \cdots \subset (W^{g})^{\prime }\subset V\oplus k$ which is $m$ -excellent such that $W^{i}=(W^{i})^{\prime }$ for $g-m\leqslant i\leqslant g$ .

3. (iii) If in part (i) both flags are $m$ -exact, then the two flags are the same.

Proof. For part (i), if $m\geqslant 1$ , then $W^{g-1}=W^{g}\cap V$ is uniquely determined. Next if $m\geqslant 2$ , then the flag is not $1$ -good and $T(W^{g-1})\not \subset \unicode[STIX]{x1D70B}_{1}(W^{g})$ . Since goodness requires $T(W^{g-2})\subset \unicode[STIX]{x1D70B}_{1}(W^{g})$ , we have $W^{g-2}=W^{g-1}\,\cap \,T^{-1}(\unicode[STIX]{x1D70B}_{1}(W^{g}))$ is also unique. Similarly if $m\geqslant 3$ , then the flag is not $2$ -good and $T(W^{g-2})\not \subset W^{g-1}$ while $T(W^{g-3})\not \subset W^{g-1}$ , and thus $W^{g-3}=W^{g-2}\,\cap \,T^{-1}(W^{g-1})$ is unique. Proceed likewise and we have the uniqueness of $W^{g-1}$ ,…, $W^{g-m}$ . This proves part (i). Now suppose the flag is $m$ -good and $(m+1)$ -good. Then $T(W^{g-m})\subset W^{g-m+1}$ and $T(W^{g-m-1})\subset W^{g-m}$ . However, the previous lemma implies $T(W^{g-m})\not \subset W^{g-m}$ . Thus $W^{g-m-1}=W^{g-m}\,\cap \,T^{-1}(W^{g-m})$ is the only possibility for this to hold, i.e. for the flag to be $(m+1)$ -good. Continue the argument and we obtain part (ii). Finally part (iii) follows directly from parts (i) and (ii).◻

Let $F_{T,m}^{\text{ex}}\subset F_{T,\text{good}}$ be projective varieties parameterizing $m$ -excellent flags. The key is as follows.

Lemma 3.8. We have $\tilde{j}(F_{T,m}^{\text{ex}})\subset \tilde{X}_{T,m}$ . Also when $m=0$ , $\tilde{j}(F_{T,0}^{\text{ex}})=\tilde{X}_{T,0}$ .

Lemma 3.9. Lemma 3.8 above implies Theorem 3.4.

Proof. The second statement in Lemma 3.8 gives the theorem when $m=0$ . We now use induction on $m$ . Let $\hat{F}_{T,m}\subset F_{T,m}^{\text{ex}}$ be the open subvariety that parameterize those flags that are $m$ -excellent and $(m-1)$ -general. We have $F_{T,m-1}\subset \hat{F}_{T,m}$ as a closed subvariety. By induction $\tilde{j}$ gives an isomorphism $F_{T,m-1}\cong \tilde{X}_{T,m-1}\,\backslash \,\tilde{X}_{T,m-2}$ , which is $(m-1)$ -dimensional. In particular, $F_{T,m-1}\subset \hat{F}_{T,m}$ are both non-empty.

On the other hand, a dimension count shows that $F_{T,\text{good}}$ has dimension at least $g$ , and $F_{T,m}^{\text{ex}}\subset F_{T,\text{good}}$ is a closed subvariety cut out by $g-m$ equations. As $\hat{F}_{T,m}$ is open in $F_{T,m}^{\text{ex}}$ , every component of $\hat{F}_{T,m}$ has dimension at least $m$ . This says that $F_{T,m}=\hat{F}_{T,m}\,\backslash \,F_{T,m-1}$ is non-empty (as a variety). Lemma 3.8 will force the image of $F_{T,m}$ under $\tilde{j}$ to be inside the $m$ -dimensional locus $\tilde{X}_{T,m}$ , and Lemma 3.7(iii) says that the dimension of the image has to be the same as the domain. Since $\dim \tilde{X}_{T,m}=m$ , we have $\tilde{j}(F_{T,m})\subset \tilde{X}_{T,m}$ is dense.

Since $F_{T,m}^{\text{ex}}$ is proper, $\tilde{j}(F_{T,m}^{\text{ex}})=\tilde{X}_{T,m}$ . We also have $\tilde{j}(F_{T,m-1}^{\text{ex}})=\tilde{X}_{T,m-1}$ by induction. By Lemma 3.7(ii), the image of $F_{T,m}^{\text{ex}}\,\backslash \,F_{T,m}$ under $\tilde{j}$ is in $\tilde{X}_{T,m-1}$ . By Lemma 3.7(i), the image of $F_{T,m}$ is disjoint from $\tilde{X}_{T,m-1}$ . Thus $\tilde{j}(F_{T,m})=\tilde{X}_{T,m}\,\backslash \,\tilde{X}_{T,m-1}$ . The proof of the uniqueness in Lemma 3.7(i) can be carefully checked to imply that not only $\tilde{j}$ is injective on closed point, but also $\tilde{j}:F_{T,m}\cong \tilde{X}_{T,m}\,\backslash \,\tilde{X}_{T,m-1}$ is an isomorphism.◻

Proof of Lemma 3.8.

Recall $\tilde{X}_{T,0}=(\times 2)^{-1}(\infty )$ by definition. By Lemma 3.2(i), $(\times 2)^{-1}(\infty )$ parameterizes $g$ -dimensional varieties $W^{g}$ in $V$ on which $\langle \cdot \,,\cdot \rangle _{1}|_{V}=\langle \cdot \,,\cdot \rangle$ and $\langle \cdot \,,\cdot \rangle _{2}|_{V}=\langle \cdot ,T\cdot \rangle$ vanish. Lemma 3.7(ii) and (iii) then says it extends uniquely to a $0$ -exact flag (there we began with a flag rather than $W^{g}$ itself, but the same proof applies). This proves the $m=0$ statement.

From now on $0<m\leqslant g$ is fixed. We shall show $\tilde{j}(F_{T,m}^{\text{ex}})\subset \tilde{X}_{T,m}$ for a generic $T\in \mathfrak{g}(1)^{\text{rs}}$ (i.e. for $T$ in a Zariski open subset of $\mathfrak{g}(1)^{\text{rs}}$ ). In fact, what we will do is the following: fix a flag $\mathbb{F}=(0\subset W^{1}\subset \cdots \subset W^{g-1}\subset W^{g}\subset V\oplus k)$ such that $W^{g-1}\subset V$ , $W^{g}\not \subset V$ , $k\not \subset W^{g}$ (here $V$ and $k$ are the first and the second components in $V\oplus k$ ), and $\langle \cdot \,,\cdot \rangle$ is trivial on $\unicode[STIX]{x1D70B}_{1}(W^{g})$ . Since all such flags in $V\oplus k$ are conjugate by $G(0)$ (where $G(0)$ preserves $V\subset V\oplus k$ and acts trivially on the second component), without loss of generality we may assume that $\mathbb{F}$ is exactly the flag in interest.

There is an irreducible closed subvariety ${\mathcal{V}}\subset \mathfrak{g}(1)$ such that $\mathbb{F}$ is $m$ -excellent with respect to $T\in \mathfrak{g}(1)^{\text{rs}}$ if and only if $T$ lies inside ${\mathcal{V}}$ . There is an Zariski open subset of ${\mathcal{V}}$ consisting of those $T$ for which the flag is $m$ -exact. What we shall prove is that for an even smaller open subset $\unicode[STIX]{x1D6E5}\subset {\mathcal{V}}$ , all $T\in \unicode[STIX]{x1D6E5}$ satisfy $\tilde{j}(\mathbb{F})\in \tilde{X}_{T,m}$ . A continuity argument by having Theorem 3.1 in family then extends the result to all $T\in {\mathcal{V}}$ , which is what we need.

The case $m=g$ is trivial, and we will assume $0<m<g$ . Let $0\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ be an $m$ -good flag. Consider $U_{0}=W^{g-m}$ and $U^{0}=(W^{g-m+1})^{\bot 1}:=\{(v,c)\in V\oplus k\mid \langle v,w\rangle =0$ , for all $w\in W^{g-m+1}\}$ (that is, $^{\bot 1}$ is used to denote the orthogonal complement with respect to $\langle \cdot \,,\cdot \rangle _{1}$ ). Following the spirit of [Wan13, § 3.1], we define inductively subspaces $U_{0}\subset U_{1}\subset \cdots \subset U_{\lfloor (m+1)/2\rfloor }\subset U^{\lfloor (m+1)/2\rfloor }\subset U^{\lfloor (m+1)/2\rfloor -1}\subset \cdots \subset U^{0}$ as follows:

$$\begin{eqnarray}\left\{\begin{array}{@{}l@{}}U_{n}:=(U^{n-1})^{\bot 1}\cap W^{g-m+2n-1},\\ U^{n}:=(U_{n})^{\bot 2}\cap U^{n-1}.\end{array}\right.\end{eqnarray}$$

Lemma 3.10. For $0\leqslant n<\lfloor (m+1)/2\rfloor$ , we have:

1. (i) $U^{n}\supset (W^{g-m+2n+1})^{\bot 1}$ .

For $n=m/2$ we have:

1. (ii) $U^{n}\supset k$ .

For $0\leqslant n\leqslant \lfloor (m+1)/2\rfloor$ we have:

1. (iii) $\dim U_{n}=g-m+n$ ;

1. (iv) $\dim U^{n}=g+m-n+1$ .

Proof. For $n=0$ it is obvious. We now do induction on $n>0$ . Let $0<n<(m+1)/2$ . Since $U_{n}\subset W^{g-m+2n-1}\subset W^{g-1}\subset V$ , we have $U^{n}\supset k$ . This in particular gives part (ii). Now suppose $n<\lfloor (m+1)/2\rfloor$ . That $\mathbb{F}$ is good (with respect to $T$ ) says $T(W^{g-m+2n-1})\subset \unicode[STIX]{x1D70B}_{1}(W^{g-m+2n+1})$ . Since $U_{n}\subset W^{g-m+2n-1}$ , from definition of $\langle \cdot \,,\cdot \rangle _{2}$ we have $(U_{n})^{\bot 2}\supset (W^{g-m+2n+1})^{\bot 1}$ . Also by induction $U^{n-1}\supset (W^{g-m+2n-1})^{\bot 1}\supset (W^{g-m+2n+1})^{\bot 1}$ . This gives part (i).

For part (iii), since $U^{n-1}\supset (W^{g-m+2n-1})^{\bot 1}$ by part (i), we have $k\subset (U^{n-1})^{\bot 1}\subset W^{g-m+2n-1}+k$ , where $k$ denotes the second component in $V\oplus k$ , i.e. the kernel of $\langle \cdot \,,\cdot \rangle _{1}$ . Since $k\not \subset W^{g-m+2n-1}$ , this says $\dim U_{n}=\dim (U^{n-1})^{\bot 1}-1=g-m+n$ . This proves part (iii).

Lastly for part (iv), we begin by observing $U^{n-1}=(U_{n-1})^{\bot 2}\cap U^{n-1}$ , i.e.  $U^{n-1}\subset (U_{n-1})^{\bot 2}$ . For $n>1$ this follows from definition, while for $n=1$ this says $(W^{g-m+1})^{\bot 1}\subset (W^{g-m})^{\bot 2}$ . By definition of $\langle \cdot \,,\cdot \rangle _{2}$ the latter is equivalent to $T(W^{g-m})\subset \unicode[STIX]{x1D70B}_{1}(W^{g-m+1})$ , i.e. that our flag is $m$ -good. Now since $U_{n}$ contains $U_{n-1}$ with codimension one by part (iii), to see that $U^{n}$ is contained in $U^{n-1}$ with codimension one it suffices to show $(U_{n})^{\bot 2}\not \supset U^{n-1}$ . Suppose on the contrary $(U_{n})^{\bot 2}\supset U^{n-1}$ , then we have $U^{n-1}\subset (U_{n})^{\bot 1}\cap (U_{n})^{\bot 2}$ . In this case we have $k\subset U^{n-1}\Rightarrow U_{n}\subset V=k^{\bot 2}$ . From the definition of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ we get $\unicode[STIX]{x1D70B}_{1}(U^{n-1})\subset (U_{n})^{\bot }\cap T(U_{n})^{\bot }$ , where this time $^{\bot }$ means the orthogonal complement in $V$ with respect to $\langle \cdot \,,\cdot \rangle$ . But this is impossible, because by Lemma 3.6 $(U_{n})^{\bot }\cap T(U_{n})^{\bot }$ intersect non-trivially and thus have dimension less than $g-m+n$ by part (iii), while $\dim \unicode[STIX]{x1D70B}_{1}(U^{n-1})=\dim U^{n-1}-1=g+m-n+1$ by inductive hypothesis from part (iv).◻

We now come back to the proof of Lemma 3.8. Define $L$ to be the variety that parameterize $g$ -dimensional subspaces $W$ satisfying $U_{\lfloor (m+1)/2\rfloor }\subset W\subset U^{\lfloor (m+1)/2\rfloor }$ and that $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish on $W$ . In particular, by construction we have $U_{\lfloor (m+1)/2\rfloor }\subset W^{g-m+2\lfloor (m+1)/2\rfloor -1}\subset W^{g}$ and $W^{g}\subset (W^{g})^{\bot 2}\cap U^{0}\subset U^{\lfloor (m+1)/2\rfloor }$ , i.e.  $[W^{g}]\in L$ .

Define $\overline{V}$ to be the subquotient $\overline{V}\,:=\,U^{\lfloor (m+1)/2\rfloor }/U_{\lfloor (m+1)/2\rfloor }$ . Since $U_{\lfloor (m+1)/2\rfloor }\subset (U^{\lfloor (m+1)/2\rfloor -1})^{\bot 1}\subset (U^{\lfloor (m+1)/2\rfloor })^{\bot 1}$ and $U^{\lfloor (m+1)/2\rfloor }\subset (U_{\lfloor (m+1)/2\rfloor })^{\bot 2}\Rightarrow U_{\lfloor (m+1)/2\rfloor }\subset (U^{\lfloor (m+1)/2\rfloor })^{\bot 2}$ , the two quadratic forms $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ restricts to be quadratic forms on  $\overline{V}$ . Denote still their restrictions by $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ , respectively. Let $L^{\prime }$ be the variety that parameterize $\lfloor m/2\rfloor$ -dimensional subspaces in $\overline{V}$ on which $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ are trivial. Then we have $\unicode[STIX]{x1D704}_{T}:L\xrightarrow[]{{\sim}}L^{\prime }$ which sends a $g$ -dimensional subspace of $V$ contained in $U^{\lfloor (m+1)/2\rfloor }$ to its image in $\overline{V}$ .

Consider the polynomials $p_{T}^{(i)}(x)=\text{disc}(\langle \cdot \,,\cdot \rangle _{1}-x\langle \cdot \,,\cdot \rangle _{2})|_{U^{i}/U_{i}}$ for $i=0,1,\ldots ,\lfloor (m+1)/2\rfloor$ . We claim $p_{T}^{(0)}(x)=x^{2\lfloor (m+1)/2\rfloor }p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ . To see this, observe that when we go from $U^{0}/U_{0}$ to $U^{1}/U_{1}$ , we quotient out $U_{1}/U_{0}$ , on which both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ is zero. Even more, $U_{1}/U_{0}$ is in the kernel of $\langle \cdot \,,\cdot \rangle _{1}$ while $\langle U_{1}/U_{0},U^{0}/U^{1}\rangle _{2}$ is non-trivial. This exactly says $p_{T}^{(0)}(x)=x^{2}p_{T}^{(1)}(x)$ . Repeating the argument gives the asserted result. We also note $\deg p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)=2m-2\lfloor (m+1)/2\rfloor +1=2\lfloor m/2\rfloor +1$ .

Let $\bar{C}_{T}$ be the hyperelliptic curve given by $(y^{2}=p_{T}^{(\lfloor (m+1)/2\rfloor )}(x))$ with completion smooth at infinity. Recall $T\in {\mathcal{V}}$ is such that our flag $\mathbb{F}$ is $m$ -excellent with respect to $T$ . One checks from definition that when $T$ runs over the subspace ${\mathcal{V}}$ , $p_{T}^{(0)}(x)$ runs over all monic polynomials of degree less than or equal to $2m+1$ that are divided by $x^{m+1}$ . Consequently, there exists a Zariski open subset $\unicode[STIX]{x1D6E5}\subset {\mathcal{V}}$ such that for $T\in \unicode[STIX]{x1D6E5}$ , $p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is separable (i.e. having distinct roots) and $\bar{C}_{T}$ is smooth.

For such $T$ , the relation between $L^{\prime }$ and $\bar{C}_{T}$ is exactly that between $(\times 2)^{-1}(\infty )$ and $C_{T}$ in Lemma 3.2(i). In particular, we have a simply transitive $\operatorname{Pic}^{0}(\bar{C}_{T})[2]$ -action on $L^{\prime }$ described by Lemma 3.3. Let $\overline{\unicode[STIX]{x1D6FA}}_{0}\subset \overline{V}$ be the $\lfloor m/2\rfloor$ -dimensional subspace that corresponds to $W^{g}$ , i.e.  $\unicode[STIX]{x1D704}_{T}([W^{g}])=[\overline{\unicode[STIX]{x1D6FA}}_{0}]$ . Let $\bar{p}_{0},\ldots ,\bar{p}_{2\lfloor m/2\rfloor },\bar{\infty }$ be the Weierstrass points of $\bar{C}_{T}$ , the last one understood as the point at infinity. We define $\overline{\unicode[STIX]{x1D6FA}}_{i}$ for $i=1,2,\ldots ,2\lfloor m/2\rfloor +1$ by $[\overline{\unicode[STIX]{x1D6FA}}_{i}]=((\bar{p}_{i-1})-(\bar{\infty })).[\overline{\unicode[STIX]{x1D6FA}}_{i-1}]$ . By Lemma 3.3, $\overline{\unicode[STIX]{x1D6FA}}_{i-1}$ and $\overline{\unicode[STIX]{x1D6FA}}_{i}$ intersect in codimension one. We note that the relation $\sum ((\bar{p}_{i})-(\bar{\infty }))=0$ implies $\overline{\unicode[STIX]{x1D6FA}}_{2\lfloor m/2\rfloor +1}=\overline{\unicode[STIX]{x1D6FA}}_{0}$ .

Now we pass this sequence from $L^{\prime }$ to $L$ . Let $\unicode[STIX]{x1D6FA}_{i}\subset V\oplus k$ be the preimage of $\overline{\unicode[STIX]{x1D6FA}}_{i}$ for each $i$ . We also have $\unicode[STIX]{x1D6FA}_{i-1}$ and $\unicode[STIX]{x1D6FA}_{i}$ intersect in codimension one for each $i=1,2,\ldots ,2\lfloor m/2\rfloor +1$ . This will allow us to fetch the precise information we want about $[\unicode[STIX]{x1D6FA}_{i}]$ using the description of the group structure of $G_{T}$ following Theorem 3.1: the subspace $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ is $(g+1)$ -dimensional, and any quadratic form among $x_{1}\langle \cdot \,,\cdot \rangle _{1}+x_{2}\langle \cdot \,,\cdot \rangle _{2}$ factor through a linear pairing of $(\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i})/\unicode[STIX]{x1D6FA}_{i-1}$ and $(\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i})/\unicode[STIX]{x1D6FA}_{i}$ since the quadratic form is trivial on both $\unicode[STIX]{x1D6FA}_{i-1}$ and $\unicode[STIX]{x1D6FA}_{i}$ by assumption. Consequently some non-trivial combination of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ must vanish on $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ . The space $\unicode[STIX]{x1D6FA}_{i-1}$ , $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ and $\unicode[STIX]{x1D6FA}_{i}$ then play the role of $W$ , $W^{\prime }$ and $W^{\prime \prime }$ in the paragraph following Theorem 3.1, respectively, which means there exists some $p_{i}\in C_{T}$ such that $(p_{i})-[\unicode[STIX]{x1D6FA}_{i-1}]=[\unicode[STIX]{x1D6FA}_{i}]$ .

Recall that $\unicode[STIX]{x1D6FA}_{2\lfloor m/2\rfloor +1}=\unicode[STIX]{x1D6FA}_{0}=W^{g}$ . Hence, we have

(3.1) $$\begin{eqnarray}((p_{1})-((p_{2})-\cdots -((p_{2\lfloor m/2\rfloor +1})-[W^{g}])\cdots \,))=[W^{g}].\end{eqnarray}$$

Put $p_{i}^{\ast }$ to be the hyperelliptic conjugate of $p_{i}$ for $i=2,4,\ldots ,2\lfloor m/2\rfloor$ and $p_{i}^{\ast }=p_{i}$ for odd $i$ , so that $(p_{i}^{\ast })-(\infty )=(-1)^{i+1}((p_{i})-(\infty ))\in J_{T}$ . Equation (3.1) then becomes

$$\begin{eqnarray}\mathop{\sum }_{i=1}^{2\lfloor m/2\rfloor +1}(p_{i}^{\ast })-2\biggl\lfloor\frac{m}{2}\biggr\rfloor(\infty )-[W^{g}]=[W^{g}]\in F_{T}\subset G_{T}\end{eqnarray}$$

or, equivalently,

(3.2) $$\begin{eqnarray}2[W^{g}]=\mathop{\sum }_{i=1}^{2\lfloor m/2\rfloor +1}(p_{i}^{\ast })-2\biggl\lfloor\frac{m}{2}\biggr\rfloor(\infty )\in \operatorname{Pic}^{1}(C_{T})\subset G_{T}\end{eqnarray}$$

which says $(\times 2)([W^{g}])\in X_{T,2\lfloor m/2\rfloor +1}$ . This concludes the proof of Lemma 3.8 when $m$ is odd.

When $m$ is even, the fact that $p_{T}^{(0)}(x)=x^{2\lfloor (m+1)/2\rfloor }p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is divisible by $x^{m+1}$ implies that $p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is divisible by $x$ . This means $(0,0)=\bar{p}_{i}$ is one of the Weierstrass point of  $\bar{C}_{T}$ . Recall $[\overline{\unicode[STIX]{x1D6FA}}_{i}]=((\bar{p}_{i})-(\infty )).[\overline{\unicode[STIX]{x1D6FA}}_{i-1}]$ . By Lemma 3.3 we know $\langle \cdot \,,\cdot \rangle _{1}$ vanish on $\overline{\unicode[STIX]{x1D6FA}}_{i-1}+\overline{\unicode[STIX]{x1D6FA}}_{i}$ , and thus $\langle \cdot \,,\cdot \rangle _{1}$ also vanish on the preimage $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ . But this says $p_{i}=\infty$ . Thus, (3.2) gives the better result $(\times 2)([W^{g}])\in X_{T,2\lfloor m/2\rfloor }=X_{T,m}$ and we are done.◻

Remark 3.11. I first learned from Thorne the idea that symmetric powers of $C_{T}$ should arise, which he observed in his unpublished work generalizing his results in [Tho13] to nilpotent orbits of two Jordan blocks in type $\mathbf{A}$ . One can check that our varieties $X_{T,1}$ and $\tilde{X}_{T,1}$ are exactly the $X_{b}$ and $\unicode[STIX]{x1D6E4}_{b}$ in the type $\mathbf{A}$ case of [Tho13, Theorem 3.7 and § 4.1], respectively, where $b$ is the image of $T$ in the GIT quotient $\mathfrak{g}(1)/\!/G(0)$ . (Strictly speaking, $\unicode[STIX]{x1D6E4}_{b}$ only deals with the case when the orbit of $T$ meets the Kostant section.) For example, if one only wants Theorem 4.1 for $m=1$ , one may replace this whole section by the type $\mathbf{A}$ result in [Tho13, Theorem 3.7]. In fact, the curves $X_{b}$ and $\unicode[STIX]{x1D6E4}_{b}$ , possessing a Lie algebra theoretic description, will be exactly what we need in § 4 for computing subregular Shalika germs. This phenomenon works equally for all types, and is demonstrated in the latter work of the author [Tsa15b, § 4], in which $\unicode[STIX]{x1D6E4}_{b}$ is equal to $\hat{{\mathcal{H}}}_{e}(\unicode[STIX]{x1D6FE})$ where $\unicode[STIX]{x1D6FE}=T$ and $e$ is a subregular nilpotent.

In particular, results of Thorne for types $\mathbf{D}$ and $\mathbf{E}$ can also be used to obtain some subregular Shalika germs for other groups of type $\mathbf{D}$ and $\mathbf{E}$ . For example, stable subregular Shalika germs (at certain half-integral depth elements) of $\mathbf{E}_{6}$ (respectively $\mathbf{E}_{7}$ , $\mathbf{E}_{8}$ ) will be given by counting points on non-hyperelliptic curves of genus $3$ (respectively $3$ , $4$ ) given in [Tho13, Theorem 3.7].

3.2 Even case

In this subsection $n=2g+2$ ; $\tilde{G}=U_{2g+2}(E/F)$ , $G=\text{GL}_{2g+2}/_{k}=\text{GL}(V)$ , $G(0)=\text{SO}(V)$ and $\mathfrak{g}(1)=\operatorname{Sym}^{2}(V)$ where $V$ is a $(2g+2)$ -dimensional non-degenerate split quadratic space. The method in this subsection is almost identical to that of the previous one, and we only list the setting, definition and results here.

We have parallel result to Theorem 3.4. Again fix $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ . We also write $p_{T}(x)\in k[x]$ the monic characteristic polynomial of $T$ . Our hyperelliptic curve $C_{T}:=(y^{2}=p_{T}(x))$ now has two points above the infinity on $\mathbb{P}^{1}$ . We shall denote these two points by $\infty ^{(1)}$ and $\infty ^{(2)}$ . They are both defined over $k$ .

Consider $L=k[x]/p_{T}(x)$ . The Weil restriction $\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ now has not only a surjective norm map $Nm\,:\,\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\rightarrow \unicode[STIX]{x1D707}_{2}$ but also a diagonal embedding $\unicode[STIX]{x1D6E5}\,:\,\unicode[STIX]{x1D707}_{2}\rightarrow \text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ . We have $\text{Stab}_{\text{O}(V)}(T)\cong \text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ , also $\text{Stab}_{G(0)}(T)\cong \ker (\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ , and lastly $J_{T}[2]\cong \text{Stab}_{G(0)}(T)/Z(G(0))\cong (\ker (\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2}))/\unicode[STIX]{x1D6E5}(\unicode[STIX]{x1D707}_{2})$ .

We denote by $\langle \cdot \,,\cdot \rangle _{1}=\langle \cdot \,,\cdot \rangle$ the standard quadratic form on $V$ , i.e. the one which is invariant by $G(0)$ . Then $\infty ^{(1)}$ and $\infty ^{(2)}$ are just the two rulings of $\langle \cdot \,,\cdot \rangle _{1}$ . Define $\langle \cdot \,,\cdot \rangle _{2}$ on $V$ by $\langle v_{1},v_{2}\rangle _{2}=\langle v_{1},Tv_{2}\rangle _{1}$ . Then the theory of pencil of quadrics says the following.

Theorem 3.12 (Wang [Wan13, Theorem 2.26]).

Let $F_{T}$ be the variety that parameterizes $g$ -dimensional subspaces of $V$ that are isotropic with respect to $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ . Then there is a commutative algebraic group structure on

$$\begin{eqnarray}G_{T}:=J_{T}\sqcup F_{T}\sqcup \operatorname{Pic}^{1}(C_{T})\sqcup F_{T}^{\prime },\end{eqnarray}$$

where $F_{T}^{\prime }$ is a copy of $F_{T}$ as an abstract variety, and $G_{T}$ satisfies:

1. (i) $G_{T}$ has component group equal to $\mathbb{Z}/4$ ; the four components above correspond to $0+4\mathbb{Z}$ , $1+4\mathbb{Z}$ , $2+4\mathbb{Z}$ and $3+4\mathbb{Z}$ , respectively;

2. (ii) the addition law on $J_{T}\sqcup \operatorname{Pic}^{1}(C_{T})$ agrees with the natural one on $\operatorname{Pic}(C_{T})/((\infty ^{(1)})+(\infty ^{(2)})=0\!)$ ;

3. (iii) the inversion map of $G_{T}$ restricts to an isomorphism $F_{T}\xrightarrow[]{{\sim}}F_{T}^{\prime }$ .

We again write the doubling map $\times 2\,:\,F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ which is étale Galois with Galois group $J_{T}[2]$ . For $0\leqslant m\leqslant g$ with $m$ even, define $j_{m}^{(1)},j_{m}^{(2)}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by

$$\begin{eqnarray}j_{m}^{(1)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\bigg(\frac{m}{2}-1\bigg)(\infty ^{(1)})-\frac{m}{2}(\infty ^{(2)}),\end{eqnarray}$$

$$\begin{eqnarray}j_{m}^{(2)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m}{2}(\infty ^{(1)})-\bigg(\frac{m}{2}-1\bigg)(\infty ^{(2)}),\end{eqnarray}$$

and we define $X_{T,m}^{(i)}$ to be the image of $j_{m}^{(i)}$ , and $\tilde{X}_{T,m}^{(i)}=(\times 2)^{-1}(X_{T,m}^{(i)})$ , $i=1,2$ .

For $0<m\leqslant g$ with $m$ odd, we define $j_{m}^{(0)},j_{m}^{(1)},j_{m}^{(2)}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by

$$\begin{eqnarray}j_{m}^{(0)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m-1}{2}(\infty ^{(1)})-\frac{m-1}{2}(\infty ^{(2)}),\end{eqnarray}$$

$$\begin{eqnarray}j_{m}^{(1)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m-3}{2}(\infty ^{(1)})-\frac{m+1}{2}(\infty ^{(2)}),\end{eqnarray}$$

$$\begin{eqnarray}j_{m}^{(2)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m+1}{2}(\infty ^{(1)})-\frac{m-3}{2}(\infty ^{(2)}),\end{eqnarray}$$

and we define $X_{T,m}^{(i)}$ to be the image of $j_{m}^{(i)}$ , and $\tilde{X}_{T,m}^{(i)}=(\times 2)^{-1}(X_{T,m}^{(i)})$ , $i=0,1,2$ .

Next, we introduce the notion of good flags. A flag of subspaces $0\subset W^{1}\subset \cdots \subset W^{g+1}\subset V$ is called good if:

1. (i) $\dim W^{i}=i$ ;

2. (ii) the restriction of $\langle \cdot \,,\cdot \rangle _{1}$ to $W^{g+1}$ is zero;

3. (iii) the restriction of $\langle \cdot \,,\cdot \rangle _{2}$ to $W^{g}$ is zero;

4. (iv) $T(W^{i})\subset W^{i+2}$ , for all $1\leqslant i\leqslant g-1$ .

For $0\leqslant m\leqslant g$ , a good flag is called $m$ -good if $T(W^{g-m})\subset W^{g-m+1}$ . Here $W^{-1}=W^{0}=\{0\}$ , i.e. good flags are automatically $g$ -good. A flag is called $m$ -excellent if it is $n$ -good for $m\leqslant n\leqslant g$ .

For $0\leqslant m\leqslant g$ , a good flag is called $m$ -general if it is not $n$ -good for any $0\leqslant n<m$ . For any $0<m\leqslant g$ , we now define the notion of $m$ -exact flags (see also Remark 3.14). Let $\{W^{r}\}_{r=1}^{g+1}$ be any $m$ -excellent and $m$ -general flag. There always exists another $m$ -excellent flag $\{U^{r}\}_{r=1}^{g+1}$ satisfying $U^{g}=W^{g}$ but $U^{g+1}\not =W^{g+1}$ if $m$ is odd, or $U^{g-1}=W^{g-1}$ , $U^{g+1}=W^{g+1}$ but $U^{g}\not =W^{g}$ if $m$ is even. We say $\{W^{r}\}_{r=1}^{g+1}$ is $m$ -exact if $\{U^{r}\}_{r=1}^{g+1}$ is also $m$ -general. Lastly, a $0$ -excellent flag is said to be $0$ -exact.

Now let $F_{T,m}^{(1)}$ be the variety that parameterize $m$ -exact flags for which $W^{g+1}$ is in the ruling $\infty ^{(1)}$ , and $F_{T,m}^{(2)}$ be the variety that parameterizes those $m$ -exact flags for which $W^{g+1}$ is in the other ruling $\infty ^{(2)}$ . We have natural maps $\tilde{j}\,:\,F_{T,m}\,:=\,F_{T,m}^{(1)}\sqcup F_{T,m}^{(2)}\rightarrow F_{T}$ by sending $\{W^{r}\}_{r=1}^{g+1}$ to $W^{g}$ .

Theorem 3.13. For $0\leqslant m\leqslant g$ , the restriction of $\tilde{j}$ to $F_{T,m}^{(1)}$ is a locally closed embedding, with image equal to

$$\begin{eqnarray}\begin{array}{@{}cc@{}}\tilde{X}_{T,m}^{(1)}\backslash (\tilde{X}_{T,m-1}^{(0)}\cup \tilde{X}_{T,m-1}^{(1)}) & \text{if }m\text{ is even},\\ \tilde{X}_{T,m}^{(0)}\backslash (\tilde{X}_{T,m-1}^{(1)}\cup \tilde{X}_{T,m-1}^{(2)}) & \text{if }m\text{ is odd},\\ \end{array}\end{eqnarray}$$

where for $F_{T,m}^{(2)}$ , we replace, in the case $m$ is even, the two superscripts $^{(1)}$ by $^{(2)}$ .

Remark 3.14. If we relax the condition of $m$ -exactness to require only $\{W^{r}\}_{r=1}^{g+1}$ to be $m$ -excellent and $m$ -general, then the image of $\tilde{j}|_{F_{T,m}^{(1)}}$ will be $\tilde{X}_{T,m}^{(1)}\backslash \tilde{X}_{T,m-1}^{(0)}$ in the even case and $\tilde{X}_{T,m}^{(0)}\backslash \tilde{X}_{T,m-1}^{(1)}$ in the odd case. However, our definition of $m$ -exactness is what one should use for orbital integrals on even ramified unitary groups in § 4.

4 Main computation

In this section we have $n=2g+1$ except for a part of § 4.3, where we will state differently. We work with the assumption that $\text{char}(k)\gg 0$ , and leave it to Appendix C to explain why this assumption may be dropped.

The nilpotent orbits ${\mathcal{O}}\in {\mathcal{O}}(0)$ of $\tilde{\mathfrak{g}}(F)$ are classified as follows: the stable orbits, just like in $\mathfrak{gl}_{2g+1}$ , are classified by partitions $\unicode[STIX]{x1D706}=(\unicode[STIX]{x1D706}_{1}^{\unicode[STIX]{x1D6FC}_{1}}\cdots \unicode[STIX]{x1D706}_{s}^{\unicode[STIX]{x1D6FC}_{s}})$ of $2g+1$ which give the sizes of the Jordan blocks, that is $\unicode[STIX]{x1D706}_{1}>\cdots >\unicode[STIX]{x1D706}_{s}$ and $\sum \unicode[STIX]{x1D6FC}_{i}\unicode[STIX]{x1D706}_{i}=2g+1$ . In such a stable orbit, the orbits are classified by

$$\begin{eqnarray}\biggl\{(d_{i})_{i=1}^{s}\,\bigg|\,\mathop{\prod }_{\unicode[STIX]{x1D706}_{i}\text{ odd}}d_{i}=(-1)^{g}N_{E/F}E^{\times }\biggr\},\end{eqnarray}$$

in which $d_{i}\in F^{\times }/N_{E/F}E^{\times }$ ( $\cong \unicode[STIX]{x1D707}_{2}$ ) if $\unicode[STIX]{x1D6FC}_{i}(\unicode[STIX]{x1D706}_{i}-1)$ is even and $d_{i}\in \unicode[STIX]{x1D70B}^{1/2}(F^{\times }/N_{E/F}E^{\times })$ (a torsor of  $\unicode[STIX]{x1D707}_{2}$ ) if $\unicode[STIX]{x1D6FC}_{i}(\unicode[STIX]{x1D706}_{i}-1)$ is odd. We will denote by $(\unicode[STIX]{x1D706},(d_{i})_{i=1}^{s})_{\tilde{\mathfrak{g}}}$ the corresponding nilpotent orbit in $\tilde{\mathfrak{g}}(F)$ .

We explain this classification. Let $\tilde{V}$ be the standard representation of $\tilde{G}/_{E}$ , i.e.  $\tilde{V}$ is a $(2g+1)$ -dimensional hermitian space over $E$ , with hermitian form $\langle \cdot \,,\cdot \rangle$ . Begin with ${\tilde{N}}\in \tilde{\mathfrak{g}}(F)$ a nilpotent element with $\unicode[STIX]{x1D6FC}_{i}$ Jordan blocks of sizes $\unicode[STIX]{x1D706}_{i}$ . There exists a unique decomposition $\tilde{V}=\bigoplus _{i=1}^{s}\tilde{V}_{i}$ such that ${\tilde{N}}$ preserves each $\tilde{V}_{i}$ , that all Jordan blocks of ${\tilde{N}}|_{\tilde{V}_{i}}$ are of size $\unicode[STIX]{x1D706}_{i}$ , and that different $\tilde{V}_{i}$ and $\tilde{V}_{j}$ are orthogonal under $\langle \cdot \,,\cdot \rangle$ . For each $i$ , by definition ${\tilde{N}}^{\unicode[STIX]{x1D706}_{i}-1}$ induces an isomorphism from $\tilde{V}_{i}/{\tilde{N}}(\tilde{V}_{i})$ to $\ker ({\tilde{N}}|_{\tilde{V}_{i}})$ . Also one has by the anti-hermitian property of ${\tilde{N}}$ that $\langle {\tilde{N}}(\tilde{V}_{i}),\ker ({\tilde{N}}|_{\tilde{V}_{i}})\rangle =0$ . This allows us to consider a pairing on $\tilde{V}_{i}/N(\tilde{V}_{i})$ by $\langle \cdot ,{\tilde{N}}^{\unicode[STIX]{x1D706}_{i}-1}\cdot \rangle$ . This pairing is non-degenerate, and it is hermitian if $\unicode[STIX]{x1D706}_{i}$ is odd and anti-hermitian if $\unicode[STIX]{x1D706}_{i}$ is even. The invariant $d_{i}$ is then the discriminant of this pairing.

On the other hand, hermitian (or anti-hermitian) pairing over local field are determined (up to isomorphism) by discriminants, and one can reconstruct $\tilde{V}$ from all of the discriminants of such pairings. With the product formula $\prod _{\unicode[STIX]{x1D706}_{i}\text{ odd}}d_{i}=(-1)^{g}N_{E/F}E^{\times }$ to ensure that $\tilde{V}$ has discriminant $1$ , such data parameterize $(\tilde{V},{\tilde{N}})$ , where $\tilde{V}$ is a hermitian space with discriminant $1$ and ${\tilde{N}}$ an anti-hermitian nilpotent operator on $\tilde{V}$ , up to conjugation by $\operatorname{End}_{E}(\tilde{V})$ . This is the same as to parameterize nilpotent ${\tilde{N}}$ up to conjugation by $\tilde{G}(F)=U(\tilde{V})$ .

Similarly, we can speak of nilpotent orbits in $\mathfrak{g}(1)(k)$ , i.e.  $G(0)(k)$ -orbit in $\mathfrak{g}(1)(k)$ that are nilpotent in $\mathfrak{g}$ . The stable orbits correspond to the same partitions, and the orbits inside a stable orbit are classified by

$$\begin{eqnarray}\Bigl\{(d_{i})_{i=1}^{s}\mid d_{i}\in k^{\times }/k^{\times 2},\prod \mathop{d}_{i}^{\unicode[STIX]{x1D706}_{i}}=(-1)^{g}\Bigr\}.\end{eqnarray}$$

We will denote by $(\unicode[STIX]{x1D706},(d_{i})_{i=1}^{s})_{\mathfrak{g}(1)}$ the corresponding nilpotent orbit in $\mathfrak{g}(1)(k)$ . The classification is done in the same way as above by replacing $\tilde{V}$ by $V$ (the standard representation of $G$ ), both hermitian and anti-hermitian forms by quadratic forms over $k$ , and both $F^{\times }/N_{E/F}E^{\times }$ , $\unicode[STIX]{x1D70B}^{1/2}(F^{\times }/N_{E/F}E^{\times })$ by $k^{\times }/k^{\times 2}$ . It is not hard to check that this set is in bijection with the previous one for nilpotent orbits of the unitary group. For our purpose we consider a canonical bijection as follows: for any $N\in \mathfrak{g}(1)(k)$ nilpotent, there exists a lift ${\tilde{N}}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ which is also nilpotent. The orbit of such ${\tilde{N}}$ is uniquely determined by the orbit of $N$ .

Let $N_{0}\in \mathfrak{g}(1)(k)$ be an arbitrary regular nilpotent element, i.e. one with a single Jordan block. For $1\leqslant m\leqslant g$ , let $N_{m}\in \mathfrak{g}(1)(k)$ be a nilpotent element with two Jordan blocks of sizes $2g+1-m$ and $m$ whose orbit is classified as $((2g+1-m,m),(-1)^{g},1)_{\mathfrak{g}(1)}$ if $m$ is even and $((2g+1-m,m),1,(-1)^{g})_{\mathfrak{g}(1)}$ if $m$ is odd. Write ${\tilde{N}}_{m}$ for the corresponding nilpotent orbit in $\tilde{\mathfrak{g}}(F)$ . For $m>0$ it is classified by $((2g+1-m,m),(-1)^{g},\unicode[STIX]{x1D70B}^{-1/2})_{\mathfrak{g}(1)}$ if $m$ is even and $((2g+1-m,m),\unicode[STIX]{x1D70B}^{-1/2},(-1)^{g})_{\mathfrak{g}(1)}$ if $m$ is odd.

When $m>0$ there are always two orbits in the stable orbit of ${\tilde{N}}_{m}$ . We again fix $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ and a lift $\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ in this section. We shall prove our main theorem (see Theorem 1.1 for the definition of $a_{m}(T)$ ).

Theorem 4.1. For $0\leqslant m\leqslant g$ , we have $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}^{\text{st}}(\tilde{T})=a_{m}(T)$ for any lift $\tilde{T}$ of $T$ .

Also $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}^{\prime }}^{\text{st}}(\tilde{T})=(-1)^{m}a_{m}(T)$ for the other nilpotent orbit ${\tilde{N}}_{m}^{\prime }$ in the same stable orbit.

For notational convenience, in this section we only compute the Shalika germs for ${\tilde{N}}_{m}$ . For the other orbit the computation is identical except that we should replace $C_{T}$ by its quadratic twist, resulting in the sign $(-1)^{m}$ in the theorem.

We want to plug in the Shalika expansion (1.1) some test functions $f$ that are locally constant by a ‘sufficiently large’ lattice and for which we know how to compute $J(\tilde{T},f)$ . Let $S\subset B\subset G$ be a choice of $\unicode[STIX]{x1D703}$ -stable maximal $k$ -torus and Borel $k$ -subgroup (see also the end of § 2.1). Let $\mathfrak{b}=\text{Lie }B$ and let $B(0)=B\cap G(0)$ , $\mathfrak{b}(i)=\mathfrak{b}\cap \mathfrak{g}(i)$ , so that $\mathfrak{b}=\mathfrak{b}(0)\oplus \mathfrak{b}(1)$ . The same notation applies to $S$ .

There exists a point $\boldsymbol{y}$ on the Bruhat–Tits building, which can be taken to be the barycenter of some alcove neighboring to $\mathbf{x}$ , such that $\tilde{\mathfrak{g}}(F)_{\boldsymbol{y},-1/2}$ is the preimage of $\mathfrak{b}(1)(k)$ under $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}{\twoheadrightarrow}\mathfrak{g}(1)(k)$ . We make the following hypothesis, which holds by [DeB02, Theorem 2.1.5] when $\text{char}(k)$ is large enough (compared with $g$ ).

Hypothesis 4.2. The Shalika expansion (1.1)

$$\begin{eqnarray}J(\tilde{T},f)=\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0)}\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T})J({\mathcal{O}},f),\end{eqnarray}$$

holds for any compactly supported function $f$ that are locally constant by $\tilde{\mathfrak{g}}(F)_{\boldsymbol{y},-1/2}$ .

Now we can choose our test functions. Fix now $0\leqslant m\leqslant g$ and let $N_{m}$ be as before. After a conjugation by some element in $G(0)(k)$ , we may and shall assume that there exists a cocharacter $\unicode[STIX]{x1D70C}_{m}\,:\,\mathbb{G}_{m}/_{k}\rightarrow S(0)$ such that $\unicode[STIX]{x1D70C}_{m}(\unicode[STIX]{x1D706})$ acts on $N_{m}$ by $\unicode[STIX]{x1D706}^{-2}$ . Write $\mathfrak{g}_{j}\subset \mathfrak{g}$ for the subspace on which $\unicode[STIX]{x1D70C}_{m}(\unicode[STIX]{x1D706})$ acts by $\unicode[STIX]{x1D706}^{j}$ , $\mathfrak{g}(1)_{j}=\mathfrak{g}_{j}\cap \mathfrak{g}(1)$ and $\mathfrak{g}(1)_{{\geqslant}i}=\bigoplus _{j\geqslant i}\mathfrak{g}(1)_{j}$ . We may and shall assume that $\unicode[STIX]{x1D70C}_{m}$ lies in the correct Weyl chamber so that $\mathfrak{b}=\mathfrak{g}(1)_{{\geqslant}0}$ .

Let $U$ be the unipotent radical of $B$ . Here $U(0)=U\cap B(0)$ is the unipotent radical of $B(0)$ . Let $f_{m}\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))$ be the function with support inside $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ defined by $f_{m}(X)=q^{-g^{2}}=(\#U(0)(k))^{-1}$ if the image of $X$ in $\mathfrak{g}(1)$ is in the affine subspace $N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ , and $f_{m}(X)=0$ otherwise. Theorem 4.1 follows evidently from Hypothesis 4.2 and the following two propositions.

Proposition 4.3. For $0\leqslant m\leqslant g$ ,

$$\begin{eqnarray}\displaystyle J^{\text{st}}(\tilde{T},f_{m}) & = & \displaystyle \mathop{\sum }_{m^{\prime }=0}^{m}a_{m^{\prime }}(T)[q^{\lfloor (m-m^{\prime }+1)/2\rfloor }\binom{g-m^{\prime }}{\lfloor \binom{m-m^{\prime }}{2}\rfloor }\nonumber\\ \displaystyle & & \displaystyle +\,\mathop{\sum }_{j=0}^{\lfloor (m-m^{\prime })/2\rfloor -1}(q^{m-m^{\prime }-j}-q^{m-m^{\prime }-j-1})\binom{g-m^{\prime }}{j} ].\nonumber\end{eqnarray}$$

Here $J^{\text{st}}(\tilde{T},f_{m})$ is the sum of $J(\tilde{T}^{\prime },f_{m})$ where $\tilde{T}^{\prime }$ runs over representatives of the orbits in the stable orbit of $\tilde{T}$ .

Proposition 4.4. For $0\leqslant m^{\prime }\leqslant m\leqslant g$ ,

$$\begin{eqnarray}J({\tilde{N}}_{m^{\prime }},f_{m})=q^{\lfloor (m-m^{\prime }+1)/2\rfloor }\binom{g-m^{\prime }}{\lfloor \binom{m-m^{\prime }}{2}\rfloor }+\mathop{\sum }_{j=0}^{\lfloor (m-m^{\prime })/2\rfloor -1}(q^{m-m^{\prime }-j}-q^{m-m^{\prime }-j-1})\binom{g-m^{\prime }}{j}.\end{eqnarray}$$

For nilpotent orbits ${\mathcal{O}}$ other than (the orbit of) ${\tilde{N}}_{m^{\prime }}$ with $0\leqslant m^{\prime }\leqslant m$ , $J({\mathcal{O}},f_{m})=0$ .

4.1 Geometric identification

The goal in this subsection is to prove Proposition 4.3. We begin with a lemma.

Lemma 4.5. There is a natural bijection between $G(0)(k)$ -orbits of $T$ in its stable orbit and $\tilde{G}(F)$ -orbit of $\tilde{T}$ in its stable orbit.

Proof. For this proof only we will replace $\tilde{G}$ by $\text{SU}_{2g+1}(E/F)$ . One checks that this replacement does not affect the orbits. Now the orbits in the stable orbit of $T$ are classified by $\ker (H^{1}(k,\text{Stab}_{G(0)}(T))\rightarrow H^{1}(k,G(0)))$ and that of $\tilde{T}$ by $\ker (H^{1}(F,\text{Stab}_{\tilde{G}}(\tilde{T}))\rightarrow H^{1}(F,\tilde{G}))$ . By Lang’s theorem and the fact that simply connected group over a non-archimedean local field has trivial $H^{1}$ , we have $H^{1}(k,G(0))=H^{1}(F,\tilde{G})=0$ . Recall also that $\text{Stab}_{G(0)}(T)\cong J_{T}[2]$ .

The key is that our $\tilde{T}$ has its centralizer $\tilde{G}_{\tilde{T}}$ is anisotropic over $F^{ur}$ , the maximal unramified extension of $F$ [Tsa15a, Theorem 2.1]. Consider the exact sequence

$$\begin{eqnarray}1\rightarrow H^{1}(\text{Gal}(F^{ur}/F),\text{Stab}_{\tilde{G}}(\tilde{T})(F^{ur}))\rightarrow H^{1}(F,\text{Stab}_{\tilde{G}}(\tilde{T}))\rightarrow H^{1}(F^{ur},\text{Stab}_{\tilde{G}}(\tilde{T})).\end{eqnarray}$$

The last cohomology group is trivial by Steinberg’s theorem. The first cohomology group is isomorphic to $H^{1}(k,J_{T}[2])$ because $J_{T}[2](\bar{k})$ is a quotient of $\text{Stab}_{\tilde{G}}(\tilde{T})(F^{ur})$ with kernel possessing a filtration with graded pieces $\cong \mathbb{G}_{a}$ .

This finishes the proof of the lemma. Note that from the exact sequence, one also sees that all orbits in the stable orbit of $\tilde{T}$ appear in $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ , and the bijection just established is compatible with the reduction map $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}{\twoheadrightarrow}\mathfrak{g}(1)(k)$ that sends $\tilde{T}\mapsto T$ .◻

The main result in this subsection is to translate the following result from § 3.1.

Lemma 4.6. For $0\leqslant m\leqslant g$ ,

$$\begin{eqnarray}J(\tilde{T},f_{m})=\frac{1}{\#J_{T}[2](k)}(\#\tilde{X}_{T,m}(k)-\#\tilde{X}_{T,m-1}(k)).\end{eqnarray}$$

and

$$\begin{eqnarray}J^{\text{st}}(\tilde{T},f_{m})=\#X_{T,m}(k)-\#X_{T,m-1}(k).\end{eqnarray}$$

Proof. To ease notation we deal with the case $m>0$ . The proof applies to $m=0$ case with a little change in various places. By [Tsa15a, Theorem 2.1], for $\tilde{h}\in \tilde{G}(F)$ , $\text{Ad}(\tilde{h})\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ if and only if $\tilde{h}\in \tilde{G}(F)_{x}$ . In particular, the centralizer $\text{Stab}_{\tilde{G}}(\tilde{T})(F)\subset \tilde{G}(F)_{x}$ . Moreover, $\tilde{G}(F)_{\mathbf{x},1/2}$ acts trivially on $f_{m}$ since $f_{m}$ is locally constant by $\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}\subset \tilde{\mathfrak{g}}(F)_{y,-1/2}$ . The integral is thus essentially a sum over $\tilde{G}(F)_{x}/\tilde{G}(F)_{\mathbf{x},1/2}\cong O_{2g+1}(k)$ .

The measure of $\tilde{G}(F)_{\mathbf{x},1/2}$ is equal to that of $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}$ , which in Appendix A can be checked to be $q^{-(2g^{2}+g)/2}$ . The measure of $\text{Stab}_{\tilde{G}}(\tilde{T})(F)_{1/2}=\text{Stab}_{\tilde{G}}(\tilde{T})(F)\cap \tilde{G}(F)_{\mathbf{x},1/2}$ is $1$ . The image of $\text{Stab}_{\tilde{G}}(\tilde{T})(F)$ in $\text{O}_{2g+1}(k)$ is equal to $\text{Stab}_{\text{O}_{2g+1}(k)}(T)$ , which has order $2\#J_{T}[2](k)$ . Also $|D(\tilde{T})|=q^{2g^{2}+g}$ . We thus have

(4.1) $$\begin{eqnarray}J(\tilde{T},f_{m})=\frac{1}{2\#J_{T}[2](k)}\mathop{\sum }_{\bar{h}\in \text{O}_{2g+1}(k)}f_{m}(\text{Ad}(\bar{h})(T))=\frac{1}{\#J_{T}[2](k)}\mathop{\sum }_{\bar{h}\in \text{SO}_{2g+1}(k)}f_{m}(\text{Ad}(\bar{h})(T)),\end{eqnarray}$$

where $f_{m}$ in the right-hand side is understood as a function on $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}\cong \mathfrak{g}(1)$ .

We have $\unicode[STIX]{x1D70C}_{m}$ acts on $V$ (the standard representation of $G(0)\cong \text{SO}_{2g+1}$ ) by weights $2g-m,2g-m-2,\ldots ,m,m-1,\ldots ,-m,-m-2,\ldots ,-2g+m$ . Let $V_{g},\ldots ,V_{-g}\subset V$ be the one-dimensional subspace on which $\unicode[STIX]{x1D70C}_{m}$ acts by scalars with corresponding weights (in order). Since the quadratic form $\langle \cdot \,,\cdot \rangle$ on $V$ is preserved by $G(0)$ , we have $V_{j}\subset V_{i}^{\bot }$ unless $i=-j$ , i.e. unless their weights sum up to zero. Note that $\unicode[STIX]{x1D70C}_{m}$ acts on $N_{m}$ with weight $-2$ . This implies $N_{m}(V_{i})=V_{i-1}$ for $m<i\leqslant g$ and $-g<i\leqslant -m$ , and that $N_{m}(V_{i+1})=V_{i-1}$ for $-m<i<m$ .

We also write $V_{{\geqslant}n}\,:=\,\bigoplus _{n\leqslant i\leqslant g}V_{i}$ . From the description of $N_{m}$ above, one sees that if $\text{Ad}(h)T\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ for some $h\in G(0)(k)$ , then there exists $W^{g}\subset V\oplus k$ such that $\unicode[STIX]{x1D70B}_{1}(W^{g})=h^{-1}V_{{\geqslant}1}$ , and the flag $(0\subset h^{-1}V_{g}\subset h^{-1}V_{{\geqslant}g-1}\subset \cdots \subset h^{-1}V_{{\geqslant}2}\subset W^{g})$ is $m$ -exact (see the definition of $m$ -exactness in the paragraph before Theorem 3.4).

In fact, by the definition of $N_{m}$ , there exists $v_{1}\in V_{1}$ such that $\langle v_{1},N_{m}(v_{1})\rangle =1$ . With it $W^{g}$ can be given by either of the following two choices $W^{g}=h^{-1}V_{{\geqslant}2}+(h^{-1}.v_{1},\pm 1)$ , where $h^{-1}V_{{\geqslant}2}$ is a subspace of $V$ and thus of $V\oplus k$ .

Conversely, if there exists an $m$ -exact flag $0\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ , then there is a unique right $B(0)(k)$ -coset in $G(0)(k)$ , say $B(0)(k)\cdot h$ , such that $W^{i}=h^{-1}V_{{\geqslant}g-i+1}$ for $1\leqslant i<g$ and $\unicode[STIX]{x1D70B}_{1}(W^{g})=h^{-1}V_{{\geqslant}1}$ . In this right coset, one checks from the definition of $m$ -exactness that there are exactly two right $U(0)(k)$ -cosets, say $U(0)(k)\cdot h$ , such that $\text{Ad}(h)T\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ .

In other words, there is a two-to-two correspondence between such $m$ -exact flags and right $U(0)(k)$ -cosets $U(0)(k)\cdot h\subset G(0)(k)$ satisfying $\text{Ad}(h)T\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ . Since $f_{m}(X)=q^{-g^{2}}$ when $X\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ (recall $q^{-g^{2}}=(\#U(0)(k))^{-1}$ ), we conclude from (4.1) that

$$\begin{eqnarray}J(\tilde{T},f_{m})=\frac{1}{\#J_{T}[2](k)}\#F_{T,m}(k)=\frac{1}{\#J_{T}[2](k)}(\tilde{X}_{T,m}(k)-\tilde{X}_{T,m-1}(k)).\end{eqnarray}$$

The last equality follows from Theorem 3.4. This proves the first statement of the lemma.

For the stable Shalika germ, by Lemma 4.5 we have $T$ running over $G(0)(k)$ -orbits in its stable orbit, classified by $H^{1}(k,J_{T}[2])$ . When $T$ runs over these orbits, above every $k$ -point of $X_{T,m}$ , all isomorphism classes of $J_{T}[2]$ -torsor will appear exactly once in $\tilde{X}_{T,m}$ . Since $\#H^{1}(k,J_{T}[2])=\#H^{0}(k,J_{T}[2])=\#J_{T}[2](k)$ , the sum of the number of $k$ -points in all isomorphism classes is exactly $\#J_{T}[2](k)$ . This gives

$$\begin{eqnarray}J^{\text{st}}(\tilde{T},f_{m})=\#X_{T,m}(k)-\#X_{T,m-1}(k).\end{eqnarray}$$

◻

It is now straightforward to verify that Proposition 4.3 follows from Lemma 4.6 and the following lemma.

Lemma 4.7. For $0\leqslant m\leqslant g$ , write $\widehat{a}_{m}(T)=(-1)^{m}\text{Tr}(\text{Frob}\,:\,H^{m}(J_{T}/_{\bar{k}},\mathbb{Q}_{\ell }))$ . We have:

1. (i) $\#X_{T,m}(k)=\#\operatorname{Sym}^{m}(C_{T})(k)-q\#\operatorname{Sym}^{m-2}(C_{T})(k)$ ;

2. (ii) $\#\operatorname{Sym}^{m}(C_{T})(k)=\sum _{m^{\prime }=0}^{m}(q^{m^{\prime }}+\cdots +q+1)\widehat{a}_{m-m^{\prime }}(T)$ ;

3. (iii) $\widehat{a}_{m}(T)=\sum _{m^{\prime }=0}^{\lfloor m/2\rfloor }q^{m^{\prime }}\binom{g-m+2m^{\prime }}{m^{\prime }}a_{m-2m^{\prime }}(T)$ .

Proof. For assertion (i), the variety $X_{T,m}\subset \operatorname{Pic}^{1}(C_{T})$ can be identified (by subtracting $(\infty )$ ) with the image ${\dot{X}}_{T,m}:=\text{im}(AJ_{m})$ of the Abel–Jacobi map $AJ_{m}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow J_{T}$ . For every rational point $\unicode[STIX]{x1D709}\in {\dot{X}}_{T,m}(k)$ represented by a divisor $D-m(\infty )$ (we may and shall assume $D$ is effective), let $d=\ell (D):=H^{0}(C_{T},D)-1$ be the dimension of the linear system $|D|$ . We claim that $\#((AJ_{m})^{-1}(\unicode[STIX]{x1D709}))(k)=q^{d}+q^{d-1}+\cdots +1$ . Note that the closed points on the fiber $(AJ_{m})^{-1}(\unicode[STIX]{x1D709})$ are effective divisors that are linear equivalent to $D$ .

Suppose $D=(p_{1})+\cdots +(p_{m^{\prime }})+(q_{1})+(q_{1}^{\ast })+\cdots +(q_{m^{\prime \prime }})+(q_{m^{\prime \prime }}^{\ast })$ , $m=m^{\prime }+2m^{\prime \prime }$ is such that $q_{i}$ and $q_{i}^{\ast }$ are hyperelliptic conjugate of each other and none of $p_{i}$ is the hyperelliptic conjugate of $p_{j}$ for any $i\not =j$ . Then one has $d=m^{\prime \prime }$ , and all effective divisors of degree $m$ that are equivalent to $D$ are of the form $D=(p_{1})+\cdots +(p_{m^{\prime }})+(\dot{q}_{1})+(\dot{q}_{1}^{\ast })+\cdots +(\dot{q}_{m^{\prime \prime }})+(\dot{q}_{m^{\prime \prime }}^{\ast })$ , where $\dot{q}_{i}$ and $\dot{q}_{i}^{\ast }$ are hyperelliptic conjugate. Pairs of hyperelliptic conjugate points on $C_{T}$ are parameterized by the base $\mathbb{P}^{1}$ . Hence, closed points on $(AJ_{m})^{-1}(\unicode[STIX]{x1D709})$ correspond to points on $\operatorname{Sym}^{d}(\mathbb{P}^{1})=\mathbb{P}^{d}$ . This proves the claim.

Note that $\unicode[STIX]{x1D709}\in {\dot{X}}_{T,m-2}=AJ_{m-2}$ if and only if $D$ is equivalent to some $(p_{1})+\cdots +(p_{m-2})+(\infty )+(\infty )$ , i.e.  $d\geqslant 1$ . In this case the same proof gives $\#((AJ_{m-2})^{-1}(\unicode[STIX]{x1D709}))(k)=q^{d-1}+\cdots +1$ . Hence, the number of $k$ -points on the fiber $(AJ_{m})^{-1}(\unicode[STIX]{x1D709})$ is always one more than $q$ times the number of $k$ -points on $(AJ_{m-2})^{-1}(\unicode[STIX]{x1D709})$ , and assertion (i) follows.

The assertions (ii) and (iii) work for any curves. In this proof let us write $V_{0}=H^{0}(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell })$ for $i=0,1,2$ , so that $H^{\ast }(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell })=V_{0}\oplus V_{1}\oplus V_{2}$ as a direct sum of graded $\mathbb{Q}_{\ell }[\text{Frob}]$ -module. Note $H^{1}(J_{T}/_{\bar{k}},\mathbb{Q}_{\ell })=H^{1}(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell })=V_{1}$ , and thus $H^{\ast }(J_{T}/_{\bar{k}},\mathbb{Q}_{\ell })=\bigwedge ^{\ast }V_{1}$ . Moreover, taking symmetric power of varieties commutes with taking $\ell$ -adic cohomology (for large $\ell$ ), where symmetric power of $H^{1}$ should be realized as symmetric power as super-vector spaces, i.e.  $\bigwedge ^{\ast }H^{1}$ . This gives $H^{\ast }(\operatorname{Sym}^{m}(C_{T})/_{\bar{k}},\mathbb{Q}_{\ell })=\bigwedge ^{\ast }V_{1}\otimes \operatorname{Sym}^{\ast }(V_{0}\oplus V_{2})=H^{\ast }(J_{T}/_{\bar{k}},\mathbb{Q}_{\ell })\otimes \operatorname{Sym}^{\ast }(V_{0}\oplus V_{2})$ . The Frobenius trace on $\operatorname{Sym}^{m^{\prime }}(V_{0}\oplus V_{2})$ is $q^{m^{\prime }}+\cdots +1$ . And assertion (ii) is simply the translation of this by the Grothendieck–Lefschetz fixed point theorem.

For assertion (iii), recall that in the introduction we wrote $\unicode[STIX]{x1D706}_{1},\unicode[STIX]{x1D706}_{1}^{\prime },\ldots ,\unicode[STIX]{x1D706}_{g},\unicode[STIX]{x1D706}_{g}^{\prime }$ the Frobenius eigenvalues on $V_{1}$ . As $H^{\ast }(J_{T}/_{\bar{k}},\mathbb{Q}_{\ell })=\bigwedge ^{\ast }V_{1}$ , we see that $(-1)^{m}\widehat{a}_{m}(T)$ is the sum of all $\binom{2g}{m}$ products of elements among choices of $m$ elements from the multi-set $\{\unicode[STIX]{x1D706}_{1},\unicode[STIX]{x1D706}_{1}^{\prime },\ldots ,\unicode[STIX]{x1D706}_{g},\unicode[STIX]{x1D706}_{g}^{\prime }\}$ . On the other hand $(-1)^{m-2m^{\prime }}a_{m-2m^{\prime }}(T)$ (see definition of $a_{m}(T)$ in the introduction) is the sum of the $2^{m-2m^{\prime }}\binom{g}{m-2m^{\prime }}$ products of elements among choices of $m-2m^{\prime }$ elements from the multi-set $\{\unicode[STIX]{x1D706}_{1},\unicode[STIX]{x1D706}_{1}^{\prime },\ldots ,\unicode[STIX]{x1D706}_{g},\unicode[STIX]{x1D706}_{g}^{\prime }\}$ for which at most one of $\unicode[STIX]{x1D706}_{i}$ and $\unicode[STIX]{x1D706}_{i}^{\prime }$ is chosen for each $i$ . For any such choice, there are $\binom{g-m+2m^{\prime }}{m^{\prime }}$ ways to choose $m^{\prime }$ more pairs of $\{\unicode[STIX]{x1D706}_{i},\unicode[STIX]{x1D706}_{i}^{\prime }\}$ that are disjoint and disjoint from previous choice. This gives exactly assertion (iii).◻

4.2 Nilpotent orbital integrals

We now prove Proposition 4.4. Nilpotent orbital integrals, just like counting points on nilpotent Springer fibers, is usually purely combinatorial. Let $B^{\text{opp}}\subset G$ be the opposite Borel to $B$ with respect to $S$ . Our essential idea here is that after applying the formula of Ranga Rao [RR72, Theorem 1], we can do ‘reduction modulo $\unicode[STIX]{x1D70B}$ ’ and arrive at an integral over $G(0)(k)$ which is left invariant by $U(0)(k)$ and right invariant by $B^{\text{opp}}(0)(k)$ . This gives a combinatorial sum over $U(0)(k)\backslash G(0)(k)/B^{\text{opp}}(0)(k)$ , which is identified with the Weyl group of $G(0)$ .

To begin our proof, the formula of Ranga Rao in our case can be formulated as follows. There exists a maximal $F$ -split torus $\tilde{S}\subset \tilde{G}$ , whose corresponding apartment contains $\mathbf{x}$ and whose reduction at $\mathbf{x}$ is equal to $S(0)\subset G(0)$ . Moreover, after conjugation we may assume that the cocharacter $\unicode[STIX]{x1D70C}_{m^{\prime }}\,:\,\mathbb{G}_{m}/_{k}\rightarrow S(0)$ corresponds to $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}\,:\,\mathbb{G}_{m}/_{F}\rightarrow \tilde{S}$ and that $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ also acts on ${\tilde{N}}_{m}^{\prime }$ by weight $-2$ .

Fix such an $\tilde{S}$ and $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ . Denote by ${\mathcal{O}}$ in this subsection the orbit of ${\tilde{N}}_{m^{\prime }}$ . Write $\tilde{\mathfrak{g}}_{i}\subset \tilde{\mathfrak{g}}$ be the subspace on which $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ acts by weight $i$ . Then with suitably normalized measure, Ranga Rao’s formula says

(4.2) $$\begin{eqnarray}J({\tilde{N}}_{m^{\prime }},f_{m})=\int _{\tilde{\mathfrak{g}}_{{\leqslant}-2}(F)\cap {\mathcal{O}}}\unicode[STIX]{x1D711}(\tilde{X})\int _{\tilde{G}(F)_{\mathbf{x},0}}f_{m}(\text{Ad}(\tilde{h})\tilde{X})\,d\tilde{h}\,d\tilde{X},\end{eqnarray}$$

where $\unicode[STIX]{x1D711}(\tilde{X})$ is an $\mathbb{R}$ -valued function on $\tilde{\mathfrak{g}}_{-2}(F)$ and the measure on the first integral is a Haar measure on $\tilde{\mathfrak{g}}_{{\leqslant}-2}$ . The space $\tilde{\mathfrak{g}}_{-2}$ can be interpreted as follows: let $\tilde{V}$ be the standard representation of $\tilde{G}/_{E}$ , i.e.  $\tilde{V}$ is a $(2g+1)$ -dimensional hermitian space over $E$ . Then $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ acts on $\tilde{V}$ with weights $(2g-m^{\prime }),(2g-m^{\prime }-2),\ldots ,m^{\prime },(m^{\prime }-1),\ldots ,1,0,-1,\ldots ,-m^{\prime },(-m^{\prime }-2),\ldots ,(-2g+m^{\prime })$ .

Denote by $\tilde{V}_{g},\ldots ,\tilde{V}_{-g}$ the one-dimensional $E$ -subspace with these weights, respectively. Let $\tilde{\mathfrak{g}}_{ij}$ , $-g\leqslant i,j\leqslant g$ be the one-dimensional $E$ -subspace of $\tilde{\mathfrak{g}}$ which maps $\tilde{V}_{j}$ to $\tilde{V}_{i}$ . One can then check

$$\begin{eqnarray}\tilde{\mathfrak{g}}_{-2}=\bigoplus _{j-i=1,|i+j|>2m^{\prime }}{\tilde{\mathfrak{g}}_{i,j}\oplus \bigoplus }_{j-i=2,|i+j|<2m^{\prime }}\tilde{\mathfrak{g}}_{i,j}\end{eqnarray}$$

and

$$\begin{eqnarray}\tilde{\mathfrak{g}}_{{<}-2}=\bigoplus _{j-i=2,|i+j|\geqslant 2m^{\prime }}{\tilde{\mathfrak{g}}_{i,j}\oplus \bigoplus }_{j-i\geqslant 3}\tilde{\mathfrak{g}}_{i,j}.\end{eqnarray}$$

Note that $\tilde{\mathfrak{g}}_{ij}$ is not defined over $F$ unless $i+j=0$ , but $\tilde{\mathfrak{g}}_{ij}+\tilde{\mathfrak{g}}_{-j,-i}$ is always defined over  $F$ . Now we fix a ‘valuation-preserving’ identification of $\tilde{u} _{ij}\,:\,\tilde{\mathfrak{g}}_{ij}\rightarrow \mathbb{G}_{a}/_{E}$ so that $\tilde{u} _{ij}^{-1}(\unicode[STIX]{x1D70B}^{-1/2})$ is not in $\tilde{\mathfrak{g}}_{ij}\cap \tilde{\mathfrak{g}}(E)_{\mathbf{x},0}$ but $\tilde{u} _{ij}^{-1}(1)$ is. Let $|\cdot |\,:\,E\rightarrow \mathbb{R}$ be the extension of the standard norm on $F$ , i.e.  $|\unicode[STIX]{x1D70B}^{-1/2}|=q^{1/2}$ . One then computes

$$\begin{eqnarray}\unicode[STIX]{x1D711}(\tilde{X})=\mathop{\prod }_{j-i=2,\,j\equiv m^{\prime }(2),\,|i+j|\leqslant 2m^{\prime }}|\tilde{u} _{ij}(\tilde{X})|,\;\tilde{X}\in \tilde{\mathfrak{g}}_{{\leqslant}-2}(F).\end{eqnarray}$$

In (4.2), if $\tilde{X}\in \tilde{\mathfrak{g}}_{{\leqslant}-2}$ is such that $|\tilde{u} _{ij}(\tilde{X})|>q^{1/2}$ , then $\tilde{X}\not \in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ and $f_{m}(\text{Ad}(\tilde{h})(\tilde{X}))=0$ . Moreover, the value of $f_{m}(\text{Ad}(\tilde{h})(\tilde{X}))$ depends only on $\tilde{u} _{ij}(\tilde{X})$ modulo ${\mathcal{O}}_{E}$ since $f_{m}$ is locally constant by $\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}$ . Let us now denote by $Z_{m^{\prime }}$ the image of $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}\cap \tilde{\mathfrak{g}}_{{\leqslant}-2}(F)\cap {\mathcal{O}}$ in $\mathfrak{g}(1)(k)$ and by $(d\unicode[STIX]{x1D707})_{m^{\prime }}$ the push-forward of the measure $\unicode[STIX]{x1D711}(\tilde{X})\,d\tilde{X}|_{\tilde{\mathfrak{g}}_{{\leqslant}-2}(F)\cap \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}}$ to the finite set $Z_{m^{\prime }}$ . Also let $dh$ be the push-forward of the measure $d\tilde{h}$ from $\tilde{G}(F)_{\mathbf{x},0}$ to $G(0)(k)$ . Then we can rewrite (4.2) as

(4.3) $$\begin{eqnarray}J(\tilde{N_{m}^{\prime }},f_{m})=\int _{X\in Z_{m^{\prime }}}\int _{G(0)(k)}f_{m}(\text{Ad}(h)X)\,dh\cdot (d\unicode[STIX]{x1D707})_{m^{\prime }}.\end{eqnarray}$$

We can similarly define $V_{g},\ldots ,V_{-g}$ as one-dimensional $k$ -subspace on which $\unicode[STIX]{x1D70C}_{m^{\prime }}$ acts by strictly decreasing weight. In fact $V_{i}$ is just the line spanned by $v_{i}$ in the previous subsection (with $m$ replaced by $m^{\prime }$ ). We can then define $u_{ij}\,:\,\mathfrak{g}(k)\rightarrow k$ in the same way, scaled so that if we write the reduction maps $\text{red}_{1}\,:\,\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}\rightarrow \mathfrak{g}(1)$ and $\text{red}_{2}\,:\,\unicode[STIX]{x1D70B}^{-1/2}{\mathcal{O}}_{E}\rightarrow k$ , then $u_{ij}(\text{red}_{1}(\tilde{X}))=\text{red}_{2}(\tilde{u} _{ij}(\tilde{X}))$ for $\tilde{X}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ .

Lemma 4.8. If $f_{m}(\text{Ad}(h)(X))\not =0$ for $X\in Z_{m}^{\prime }$ , then $u_{ij}(X)\not =0$ when $j-i=2$ and $|i+j|<2m^{\prime }$ .

Proof. The function $f_{m}(X)$ , as a function of $X\in \mathfrak{g}(1)(k)$ , is invariant under conjugation by $U(0)(k)$ . The assertion of the lemma, for $X\in Z_{m}^{\prime }$ , is a property that is preserved under conjugation by $B^{\text{opp}}(0)(k)$ . Therefore, it suffices to consider $h$ in a set of representative for $U(0)(k)\backslash G(0)(k)/B^{\text{opp}}(0)(k)$ , which can be taken to be the Weyl group $N_{G(0)(k)}(S(0)(k))/S(0)(k)$ .

Identify $S_{g}\ltimes \{\pm 1\}^{g}$ with $N_{G(0)(k)}(S(0)(k))/S(0)(k)$ in the following way: the first component $S_{g}$ shall permute $V_{g},\ldots ,V_{1}$ , and the $i$ th $\{\pm 1\}$ in the second component switches $V_{i}$ and $V_{-i}$ . We now check directly the assertions for all $\unicode[STIX]{x1D70E}\in S_{g}\ltimes \{\pm 1\}^{g}$ . To have $f_{m}(\text{Ad}(\unicode[STIX]{x1D70E}^{-1})X)\not =0$ for some $X\in Z_{m^{\prime }}$ , it is necessary that $\text{Ad}(\unicode[STIX]{x1D70E})N_{m}\in Z_{m^{\prime }}$ . This happens exactly when the following condition holds.

Condition 4.9. Consider $\unicode[STIX]{x1D70E}\in S_{g}\ltimes \{\pm 1\}^{g}$ acting on $\{0,\pm 1,\ldots ,\pm g\}$ where $S_{g}$ permutes $\{1,\ldots ,g\}$ and $\{-1,\ldots ,-g\}$ simultaneously, the $i$ th component in $\{\pm 1\}^{g}$ switches $\pm i$ , and $0$ is always fixed. Now for any $-g\leqslant i<j\leqslant g$ :

1. (i) for $j-i=2$ , $|i+j|<2m$ , either $\unicode[STIX]{x1D70E}(j)-\unicode[STIX]{x1D70E}(i)=1$ and $|\unicode[STIX]{x1D70E}(i)+\unicode[STIX]{x1D70E}(j)|>2m^{\prime }$ , or $\unicode[STIX]{x1D70E}(j)-\unicode[STIX]{x1D70E}(i)\geqslant 2$ ;

2. (ii) for $j-i=1$ , $2m<|i+j|\leqslant 2g+1$ , the same condition is required.

It is straightforward to see that the condition is satisfied only when $\unicode[STIX]{x1D70E}\in S_{g}$ , i.e.  $\unicode[STIX]{x1D70E}$ preserves $\{1,\ldots ,g\}$ . One then see inductively that $\unicode[STIX]{x1D70E}^{-1}(0)=0\Rightarrow \unicode[STIX]{x1D70E}^{-1}(1)=1\Rightarrow \unicode[STIX]{x1D70E}^{-1}(2)=2\Rightarrow \cdots \,$ , until $\unicode[STIX]{x1D70E}^{-1}(m^{\prime })=m^{\prime }$ . We conclude that $\unicode[STIX]{x1D70E}$ and thus $\unicode[STIX]{x1D70E}^{-1}$ preserves $V_{1},\ldots ,V_{m^{\prime }}$ . Since $u_{i-1,i+1}(N_{m})\not =0$ for $i=0,\ldots ,m^{\prime }-1$ , this implies that $u_{i-1,i+1}(X)\not =0$ for the same $i$ , which is what we have to prove.◻

On the subset $Z_{m^{\prime }}^{o}\subset Z_{m^{\prime }}$ where the conclusion of the lemma holds, $(d\unicode[STIX]{x1D707})_{m^{\prime }}$ is nothing but a multiple of the counting measure. We will pretend it is exactly the counting measure and discuss the normalization constant later. The idea in the lemma can now be further applied to compute the integral; write

$$\begin{eqnarray}I_{m^{\prime }}(h):=\mathop{\sum }_{X\in Z_{m^{\prime }}^{o}}f_{m}(\text{Ad}(h)X).\end{eqnarray}$$

We have to compute $\sum _{h\in G(0)(k)}I_{m^{\prime }}(h)$ (up to a normalizing constant). Exactly as in the situation of the previous lemma, This function $I_{m^{\prime }}$ is invariant under left translation by $U(0)(k)$ and right translation by $B^{\text{opp}}(0)(k)$ , and we arrive at a sum over the Weyl group $S_{g}\ltimes \{\pm 1\}^{g}$ . To have $I_{m^{\prime }}(\unicode[STIX]{x1D70E}^{-1})\not =0$ , $\unicode[STIX]{x1D70E}$ needs to satisfy Condition 4.9.

Denote by $\unicode[STIX]{x1D6EF}_{m,m^{\prime }}\subset S_{g}$ the set of such $\unicode[STIX]{x1D70E}$ . For $\unicode[STIX]{x1D70E}\in \unicode[STIX]{x1D6EF}_{m,m^{\prime }}$ , the number of elements in the double coset $U(0)(k)\unicode[STIX]{x1D70E}^{-1}B^{\text{opp}}(0)(k)$ is given by $\#B(0)(k)\cdot q^{\unicode[STIX]{x1D6FF}_{1}(\unicode[STIX]{x1D70E})}$ . Also the sum $I_{m^{\prime }}(\unicode[STIX]{x1D70E}^{-1})=q^{\unicode[STIX]{x1D6FF}_{2}(\unicode[STIX]{x1D70E})-g^{2}}$ , where

$$\begin{eqnarray}\displaystyle & \displaystyle \unicode[STIX]{x1D6FF}_{1}(\unicode[STIX]{x1D70E})=g^{2}-\#\{1\leqslant i<j\leqslant g\mid \unicode[STIX]{x1D70E}(i)>\unicode[STIX]{x1D70E}(j)\}, & \displaystyle \nonumber\\ \displaystyle & \displaystyle \unicode[STIX]{x1D6FF}_{2}(\unicode[STIX]{x1D70E})=m-m^{\prime }+\#\{1\leqslant i<j\leqslant g\mid \unicode[STIX]{x1D70E}(i)>\unicode[STIX]{x1D70E}(j),\;j-i>1\}. & \displaystyle \nonumber\end{eqnarray}$$

We also have to figure out the normalization of measures. The single choice $X=N_{m^{\prime }}\in Z_{m^{\prime }}^{o}\subset \mathfrak{g}(1)(k)$ and $h=id\in G(0)(k)$ correspond to the lattice $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}/(\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}\cap \tilde{\mathfrak{g}}_{{\tilde{N}}_{m^{\prime }}})\subset \tilde{\mathfrak{g}}(F)/\tilde{\mathfrak{g}}_{{\tilde{N}}_{m^{\prime }}}(F)$ . A careful inspection of the normalization at the end of Appendix A shows that this lattice is to have measure $1$ .

In addition, as Ranga Rao’s method begins with Iwasawa decomposition $\tilde{G}=\tilde{B}\cdot \tilde{G}(F)_{\mathbf{x},0}$ , we also have to divide by how much they intersect. That is, we have to divide by the order of $(\tilde{G}(F)_{\mathbf{x},0}\cap \tilde{B})\tilde{G}(F)_{\mathbf{x},1/2}/\tilde{G}(F)_{\mathbf{x},1/2}$ , which is $\#B(0)(k)$ . In summary, all the way from (4.3) we have

(4.4) $$\begin{eqnarray}\displaystyle J({\tilde{N}}_{m^{\prime }},f_{m}) & = & \displaystyle \mathop{\sum }_{\unicode[STIX]{x1D70E}\in \unicode[STIX]{x1D6EF}_{m,m^{\prime }}}q^{\unicode[STIX]{x1D6FF}_{1}+\unicode[STIX]{x1D6FF}_{2}-g^{2}}\nonumber\\ \displaystyle & = & \displaystyle \mathop{\sum }_{\unicode[STIX]{x1D70E}\in \unicode[STIX]{x1D6EF}_{m,m^{\prime }}}q^{m-m^{\prime }-\#\{1\leqslant i<g\mid \unicode[STIX]{x1D70E}(i)>\unicode[STIX]{x1D70E}(i+1)\}}.\end{eqnarray}$$

Proposition 4.4 now follows from Proposition B.7. This completes the proof of Theorem 4.1.

4.3 More Shalika germs

Recall that in the beginning of § 3 we had the maps $j_{m}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ and $\times 2\,:\,F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ . The latter map is étale Galois with Galois group $J_{T}[2]$ . Denote by $\widetilde{\operatorname{Sym}}^{m}(C_{T}):=\operatorname{Sym}^{m}(C_{T})\times _{\operatorname{Pic}^{1}(C_{T})}F_{T}$ the fiber product, which is an étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T})$ . We have the following result.

Theorem 4.10. For $0\leqslant m\leqslant g$ , we have

$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T}) & = & \displaystyle \frac{1}{\#J_{T}[2](k)}\bigg(\mathop{\sum }_{0\leqslant 2\ell \leqslant m}\#\widetilde{\operatorname{Sym}}^{m-2\ell }(C_{T})(k)\cdot q^{\ell }\cdot C_{\ell }(-g+m-2\ell +1)\nonumber\\ \displaystyle & & \displaystyle -\,(q+1)\mathop{\sum }_{0<2\ell +1\leqslant m}\#\widetilde{\operatorname{Sym}}^{m-2\ell -1}(C_{T})(k)\cdot q^{\ell }\cdot C_{\ell }(-g+m-2\ell )\!\bigg).\nonumber\end{eqnarray}$$

See Definition B.1 for the combinatorial numbers $C_{\ell }(\cdot )$ . When $m>0$ , for the other nilpotent orbit in the stable orbit of ${\tilde{N}}_{m}$ , simply change $C_{T}$ to its quadratic twist.

Proof. Write in this proof

$$\begin{eqnarray}\vec{u}=\left(\begin{array}{@{}c@{}}\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{0}}(\tilde{T})\\ \unicode[STIX]{x1D6E4}_{{\tilde{N}}_{1}}(\tilde{T})\\ \unicode[STIX]{x1D6E4}_{{\tilde{N}}_{2}}(\tilde{T})\\ \cdots \end{array}\right),\quad \vec{v}=\left(\begin{array}{@{}c@{}}J(\tilde{T},f_{0})\\ J(\tilde{T},f_{1})\\ J(\tilde{T},f_{2})\\ \cdots \end{array}\right),\quad \vec{w}=\left(\begin{array}{@{}c@{}}\#\widetilde{\operatorname{Sym}}^{0}(C_{T})(k)\\ \#\widetilde{\operatorname{Sym}}^{1}(C_{T})(k)\\ \#\widetilde{\operatorname{Sym}}^{2}(C_{T})(k)\\ \cdots \end{array}\right).\end{eqnarray}$$

We have to write $\vec{u}$ in terms of $\vec{w}$ . The first half of Lemma 4.6 says

$$\begin{eqnarray}J_{\tilde{T}}(f_{m})=\frac{1}{\#J_{T}[2](k)}(\#\tilde{X}_{T,m}(k)-\#\tilde{X}_{T,m-1}(k))\end{eqnarray}$$

and a $J_{T}[2]$ -cover version of Lemma 4.7(i) gives

$$\begin{eqnarray}\#\tilde{X}_{T,m}(k)=\#\widetilde{\operatorname{Sym}}^{m}(C_{T})(k)-q\cdot \widetilde{\operatorname{Sym}}^{m-2}(C_{T})(k).\end{eqnarray}$$

Putting together, they imply $\vec{v}=B^{(1)}B^{(2)}\vec{w}$ , where $(B_{ij}^{(1)})_{0\leqslant i,j\leqslant g}$ and $(B_{ij}^{(2)})_{0\leqslant i,j\leqslant g}$ are lower triangular matrices with

$$\begin{eqnarray}B_{ij}^{(1)}=\left\{\begin{array}{@{}ll@{}}1 & \text{if }i=j,\\ -1 & \text{if }i=j+1,\\ 0 & \text{otherwise,}\end{array}\right.\quad B_{ij}^{(2)}=\left\{\begin{array}{@{}ll@{}}1 & \text{if }i=j,\\ -q & \text{if }i=j+2,\\ 0 & \text{otherwise}.\end{array}\right.\end{eqnarray}$$

To recover $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})$ from $J_{\tilde{T}}(f_{m})$ , i.e. compute $\vec{u}$ in terms of $\vec{v}$ , we need to ‘invert’ Proposition 4.4. One observe that Proposition 4.4 is the same as saying $\vec{v}=B^{(3)}B^{(2)}A\vec{u}$ , where $A$ is the matrix in Proposition B.6, with $x=g$ , and

$$\begin{eqnarray}(B^{(3)})_{ij}=\left\{\begin{array}{@{}ll@{}}q^{i-j} & \text{if }i\geqslant j,\\ 0 & \text{otherwise}.\end{array}\right.\end{eqnarray}$$

Proposition B.6 says

$$\begin{eqnarray}(A^{-1})_{ij}=\left\{\begin{array}{@{}ll@{}}q^{\ell }C_{\ell }(-g+j) & \text{if }i=j+2\ell ,\;\ell \in \mathbb{Z}_{{\geqslant}0},\\ 0 & \text{otherwise}.\end{array}\right.\end{eqnarray}$$

And we have $\vec{u}=A^{-1}(B^{(2)})^{-1}(B^{(3)})^{-1}B^{(1)}B^{(2)}\vec{w}$ . One observe that $B^{(1)}$ , $B^{(2)}$ and $B^{(3)}$ all commutes, and $B^{(4)}:=(B^{(3)})^{-1}(B^{(1)})$ is given by

$$\begin{eqnarray}(B^{(4)})_{ij}=\left\{\begin{array}{@{}ll@{}}1 & \text{if }i=j,\\ -(q+1) & \text{if }i=j+1,\\ q & \text{if }i=j+2,\\ 0 & \text{otherwise}.\end{array}\right.\end{eqnarray}$$

We have $\vec{u}=A^{-1}B^{(4)}\vec{w}$ . By the formula for $A^{-1}$ and $B^{(4)}$ , we see that if $i=j+2\ell$ , $\ell \in \mathbb{Z}_{{\geqslant}0}$ , then $(A^{-1}B^{(4)})_{ij}=q^{\ell }C_{\ell }(-g+j)+q^{\ell }C_{\ell -1}(-g+j+2)=q^{\ell }C_{\ell }(-g+j+1)$ by Proposition B.3, with $C_{-1}(\cdot )$ understood to be zero. And if $i=j+2\ell +1$ , $\ell \in \mathbb{Z}_{{\geqslant}0}$ , then $(A^{-1}B^{(4)})_{ij}=-(q+1)q^{\ell }C_{\ell }(-g+j+1)$ . This is essentially the content of Theorem 4.10.◻

Another way of thinking of these covers $\widetilde{\operatorname{Sym}}^{m}(C_{T})$ is as follows. Denote by $\unicode[STIX]{x1D6FC}_{T}$ the $J_{T}[2]$ -torsor $(\times 2)^{-1}(\infty )$ . Then $\widetilde{\operatorname{Sym}}^{m}(C_{T})$ is the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T})$ for which the fiber above $m(\infty )$ is isomorphic to $\unicode[STIX]{x1D6FC}_{T}$ . There is a unique such one since $J_{T}[2]$ is the maximal abelian $2$ -annihilated quotient of $\unicode[STIX]{x1D70B}_{1}^{\acute{\text{e}}\text{t},\text{tame}}(\operatorname{Sym}^{m}(C_{T}))$ .

In the rest of this section we suppose $n=2g+2$ . So that $\tilde{G}=U_{2g+2}(E/F)$ is instead an even quasi-split unitary group (still ramified). Recall we can take $\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ to be any lift of $T\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}$ , which is regular semisimple and is associated to the genus $g$ (projective smooth) hyperelliptic curve $C_{T}=(y^{2}=(-1)^{g+1}p_{T}(x))$ where $p_{T}(x)$ is the degree $2g+2$ monic characteristic polynomial of $T$ .

We use then Theorem 3.13 instead of Theorem 3.4 to obtain the following result. Let $\unicode[STIX]{x1D706}_{1},\unicode[STIX]{x1D706}_{1}^{\prime },\ldots ,$ $\unicode[STIX]{x1D706}_{g},\unicode[STIX]{x1D706}_{g}^{\prime }$ again be the Frobenius eigenvalues on $H^{1}(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell })$ so that $\unicode[STIX]{x1D706}_{i}\unicode[STIX]{x1D706}_{i}^{\prime }=q$ . We also put artificially that $\unicode[STIX]{x1D706}_{0}=1$ , $\unicode[STIX]{x1D706}_{0}^{\prime }=q$ . Write this time (note the difference on the range of $S$ with the odd case)

$$\begin{eqnarray}a_{m}(T):=(-1)^{m}\cdot \mathop{\sum }_{S\subset \{0,\ldots ,g\},|S|=m}\biggl(\mathop{\prod }_{i\in S}(\unicode[STIX]{x1D706}_{i}+\unicode[STIX]{x1D706}_{i}^{\prime })\biggr).\end{eqnarray}$$

For $0\leqslant m\leqslant g+1$ , let ${\tilde{N}}_{m}$ be any element in any nilpotent orbit in $\tilde{\mathfrak{g}}=\text{Lie }\tilde{G}$ with two Jordan blocks of sizes $2g+2-m$ and $m$ (a regular nilpotent if $m=0$ ). There can be either one, two or four of such orbits.

What we can show in parallel to Theorem 4.1, using the method in this section, is as follows.

Theorem 4.11. The stable Shalika germs at $\tilde{T}$ for nilpotent orbits with two Jordan blocks of an even quasi-split ramified unitary groups is

$$\begin{eqnarray}\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}^{\text{st}}(\tilde{T})=a_{m}(T).\end{eqnarray}$$

As for general (non-stable) Shalika germs, we encountered a technical difficulty: what was developed in Theorem 3.13 only allows us to compute Shalika germs for nilpotent $\tilde{G}^{\text{ad}}(F)$ -orbits, where $\tilde{G}^{\text{ad}}=PU_{2g+2}(E/F)$ . The image of $\tilde{G}(F)\rightarrow \tilde{G}^{\text{ad}}(F)$ has index $2$ in $\tilde{G}^{\text{ad}}(F)$ . If we take $u\in \tilde{G}^{\text{ad}}(F)$ to be any element outside the image, then what we can compute is the sum of Shalika germs $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})+\unicode[STIX]{x1D6E4}_{\text{ad}(u){\tilde{N}}_{m}}(\tilde{T})=\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})+\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\text{ad}(u)\tilde{T})$ .

To state what we are able to obtain in parallel with Theorem 4.10 with the geometry from Theorem 3.13, we need a notion about nilpotent orbits of $\tilde{G}=U_{2g+2}(E/F)$ with two even Jordan blocks.

Definition 4.12. Let $((2g+2-m,m),d_{1},d_{2})$ for $0<m<g+1$ with $m$ even (respectively $((g+1,g+1),d)$ for $m=g+1$ if $g+1$ is even) be a nilpotent orbit with two even Jordan blocks. We say the orbit is hyperbolic if $d_{1}d_{2}=-1$ (respectively $d=1$ ), and elliptic otherwise.

We also say any nilpotent orbit with two odd Jordan blocks is hyperbolic. They are characterized by the following: for any two distinct nilpotent orbits, both having two Jordan blocks, one lies in the closure of the other if and only if they have different dimensions and they are either both hyperbolic or both elliptic.

As in Theorem 4.10, we also need notation about the covers of $C_{T}$ . Recall we have two rational points $\infty ^{(1)},\infty ^{(2)}\in C_{T}(k)$ (see § 3.2). Fix a choice of any of them, say $\infty ^{(1)}$ . Consider $(\times 2)^{-1}(\infty ^{(1)})$ , where $\times 2$ is the étale $J_{T}[2]$ -Galois map in Theorem 3.12. This is a $J_{T}[2]$ -torsor, which we shall denote by $\unicode[STIX]{x1D6FC}_{T}$ .

Consider also $\infty ^{(1)}-\infty ^{(2)}\in J_{T}(k)$ . We have, by Lang’s theorem, $J_{T}(k)/2J_{T}(k)\cong H^{1}(k,J_{T}[2])$ . Denote by $\unicode[STIX]{x1D6FD}_{T}$ the $J_{T}[2]$ -torsor that are given by $\infty ^{(1)}-\infty ^{(2)}$ in this way. In fact, $(\infty ^{(1)})-(\infty ^{(2)})\in 2J_{T}(k)$ except when all irreducible factors of $p_{T}(x)\in k[x]$ are even and $n=2g+2$ is divisible by four. Consequently, if there is an odd factor of $p_{T}(x)$ or if $g$ is even, $\unicode[STIX]{x1D6FD}_{T}$ is trivial.

For even non-negative integers $m$ , we write $\widetilde{\operatorname{Sym}}^{m}(C_{T})$ the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T})$ for which the fiber above $(m/2)(\infty ^{(1)})+(m/2)(\infty ^{(2)})$ is (as a $J_{T}[2]$ -torsor) isomorphic to $\unicode[STIX]{x1D6FC}_{T}$ . Write also $\widetilde{\operatorname{Sym}}^{m,\ast }(C_{T})$ the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T})$ for which the fiber above $(m/2)(\infty ^{(1)})+(m/2)(\infty ^{(2)})$ is isomorphic to $\unicode[STIX]{x1D6FC}_{T}\times ^{J_{T}[2]}\unicode[STIX]{x1D6FD}_{T}$ .

For odd $m$ instead, we write $\widetilde{\operatorname{Sym}}^{m}(C_{T})$ the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T})$ for which the fiber above $((m+1)/2)(\infty ^{(1)})+((m-1)/2)(\infty ^{(2)})$ is isomorphic to $\unicode[STIX]{x1D6FC}_{T}$ . And we write $\widetilde{\operatorname{Sym}}^{m,\ast }(C_{T})$ for the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T})$ for which the fiber above $((m+1)/2)(\infty ^{(1)})+((m-1)/2)(\infty ^{(2)})$ is isomorphic to $\unicode[STIX]{x1D6FC}_{T}\times ^{J_{T}[2]}\unicode[STIX]{x1D6FD}_{T}$ . For all $m$ , we write $\tilde{S}^{m}(C_{T})=(\#\widetilde{\operatorname{Sym}}^{m}(C_{T})(k)+\#\widetilde{\operatorname{Sym}}^{m,\ast }(C_{T})(k))$ .

Lastly, we write $C_{T}^{\prime }$ to be the quadratic twists of $C_{T}$ , so it has two points above infinity $\infty ^{(1)}$ , $\infty ^{(2)}$ that are not defined over $k$ . Note $\operatorname{Pic}^{0}(C_{T}^{\prime })[2]\cong \operatorname{Pic}^{0}(C_{T})[2]=J_{T}[2]$ . For $m$ even, write $\widetilde{\operatorname{Sym}}^{m}(C_{T}^{\prime })$ the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T}^{\prime })$ for which the fiber above $(m/2)(\infty ^{(1)})+(m/2)(\infty ^{(2)})$ is isomorphic to $\unicode[STIX]{x1D6FC}_{T}$ . We also write $\widetilde{\operatorname{Sym}}^{m,\ast }(C_{T}^{\prime })$ for the étale $J_{T}[2]$ -cover of $\operatorname{Sym}^{m}(C_{T}^{\prime })$ for which the fiber above $(m/2)(\infty ^{(1)})+(m/2)(\infty ^{(2)})$ is isomorphic to $\unicode[STIX]{x1D6FC}_{T}\times ^{J_{T}[2]}\unicode[STIX]{x1D6FD}_{T}$ . And we write $\tilde{S}^{m}(C_{T}^{\prime })=(\#\widetilde{\operatorname{Sym}}^{m}(C_{T}^{\prime })(k)+\#\widetilde{\operatorname{Sym}}^{m,\ast }(C_{T}^{\prime })(k))$ .

Note $\#(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2})(k)=2^{r}$ where $r$ is the number of irreducible factors of $p_{T}(x)$ in $k[x]$ . We have the following result.

Theorem 4.13. For $0\leqslant m\leqslant g+1$ , let ${\tilde{N}}_{m}$ be any nilpotent orbit with two Jordan blocks of sizes $2g+2-m$ and $m$ . Recall $u\in \tilde{G}^{\text{ad}}(F)$ is any element that does not come from $\tilde{G}(F)$ . We have that if ${\tilde{N}}_{m}$ is hyperbolic, then

$$\begin{eqnarray}\displaystyle \displaystyle \unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})+\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\text{ad}(u)\tilde{T}) & = & \displaystyle \displaystyle 2^{-(r-1)}\cdot \bigg(\mathop{\sum }_{0\leqslant 2\ell \leqslant m}\tilde{S}^{m-2\ell }(C_{T})\cdot q^{\ell }\cdot \bigg(C_{\ell }(-g+m-2\ell +1)\nonumber\\ \displaystyle & & \displaystyle -\bigg(\sqrt{q}+\frac{1}{\sqrt{q}}\bigg)^{2}C_{\ell -1}(-g+m-2\ell +1)\bigg)\nonumber\\ \displaystyle & & \displaystyle \displaystyle -\,2(q+1)\mathop{\sum }_{0<2\ell +1\leqslant m}\tilde{S}^{m-2\ell -1}(C_{T})\cdot q^{\ell }\cdot C_{\ell }(-g+m-2\ell )\bigg).\nonumber\end{eqnarray}$$

If otherwise ${\tilde{N}}_{m}$ is elliptic, then

$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})+\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\text{ad}(u)\tilde{T}) & = & \displaystyle 2^{-(r-1)}\cdot \bigg(\mathop{\sum }_{0\leqslant 2\ell \leqslant m}\tilde{S}^{m-2\ell }(C_{T}^{\prime })\cdot q^{\ell }\cdot \bigg(C_{\ell }(-g+m-2\ell +1)\nonumber\\ \displaystyle & & \displaystyle +\bigg(\sqrt{q}+\frac{1}{\sqrt{q}}\bigg)^{2}C_{\ell -1}(-g+m-2\ell +1)\bigg)\bigg).\nonumber\end{eqnarray}$$

Here we adapt the convention that $\widetilde{\operatorname{Sym}}^{-1}=\widetilde{\operatorname{Sym}}^{-2}=\emptyset$ . For case $m=0$ the two formulas agree.

5 Endoscopic transfer of nilpotent orbits for ramified unitary groups

In the beginning of this section and § 5.2, i.e. except in § 5.1, we will assume $\text{char}(F)=0$ so that the endoscopic transfer of Langlands and Shelstad [LS87] is valid. We however note that we can also work with sufficiently large $\text{char}(F)$ (in an uneffective manner) thanks to Gordan and Hales [GH16].

In [Ass98], Assem stated a conjecture regarding stable distributions supported on the nilpotent cone for a reductive $p$ -adic group. Recall that a distribution ${\mathcal{D}}\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))^{\ast }$ is called stable if ${\mathcal{D}}(f)=0$ for every $f\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))$ with the property that $J^{\text{st}}(\tilde{X},f)=0$ for all $\tilde{X}\in \tilde{\mathfrak{g}}^{\text{rs}}(F)$ .

For quasi-split unitary groups, Assem’s conjecture asserts that all stable distributions supported on the nilpotent cone can be written into a linear combination of stable distributions where each term is a linear combination of nilpotent orbital integrals on various orbits in a single stable orbit. Moreover, on each stable nilpotent orbit there is a unique (up to constant) linear combination of the orbits for which the integral becomes stable.

Assem also had a conjecture regarding endoscopic transfer of nilpotent orbits. For endoscopic transfer of unitary groups, relevant elliptic endoscopy groups are products of two quasi-split unitary groups $U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)$ , while the target of endoscopy is $U_{n_{1}+n_{2}}(E/F)$ . Here the three unitary groups split over the same quadratic extension.

Recall that if $\mathbf{H}$ (e.g.  $\mathbf{H}=U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)$ ) is an endoscopy group for $\tilde{G}$ with $\tilde{\mathfrak{h}}=\text{Lie }\mathbf{H}$ , then the transfer conjecture (for the Lie algebra) asserts that for any $f\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))$ , there exists a function $f^{\mathbf{H}}\in C_{c}^{\infty }(\tilde{\mathfrak{h}}(F))$ such that

$$\begin{eqnarray}\mathop{\sum }_{{\tilde{Y}}\sim \tilde{X}}\unicode[STIX]{x1D705}({\tilde{Y}})J({\tilde{Y}},f)=J^{\text{st}}(\tilde{X},f^{\mathbf{H}}),\quad \forall \tilde{X}\in \tilde{\mathfrak{h}}^{\tilde{G}\text{-}\text{rs}}(F).\end{eqnarray}$$

Here $\tilde{X}$ is $\tilde{G}$ -regular [Wal97, 2.2] and ${\tilde{Y}}$ runs over regular semisimple orbits in $\tilde{\mathfrak{g}}(F)$ that ‘matches’ with $Y$ . Also $\unicode[STIX]{x1D705}=\unicode[STIX]{x1D705}_{\mathbf{H}}$ is some character (determined by $\mathbf{H}$ ) on the set of orbits of such  ${\tilde{Y}}$ .

The transfer conjecture was proved by Waldspurger [Wal97] conditional on Ngô’s later marvelous proof [Ngô10] of the fundamental lemma. Given the transfer conjecture, for any stable distribution ${\mathcal{D}}$ on $C_{c}^{\infty }(\mathfrak{h}(F))$ , we can define its endoscopic transfer to be the distribution ${\mathcal{D}}^{\tilde{G}}\,:\,f\mapsto {\mathcal{D}}(f^{\mathbf{H}})$ . It is obvious that such distributions has to be $\tilde{G}(F)$ -conjugation invariant.

If ${\mathcal{D}}$ is a stable distribution supported on the nilpotent cone of $\mathfrak{h}$ , i.e. it is a linear combination of nilpotent orbital integral that becomes stable, then ${\mathcal{D}}^{\tilde{G}}$ has to be also supported on the nilpotent cone. It thus makes sense to talk about endoscopic transfer of nilpotent orbital integrals.

In [Wal01], assuming $p$ large enough, Waldspurger completed the study of stability and endoscopic transfer (non-twisted endoscopy, i.e. those in [LS87]) for nilpotent orbital integrals for unramified classical groups. In particular, Assem’s conjectures (see e.g. Conjectures 5.1 and 5.5) were proved in these cases.

The endoscopy data and the transfer factor, etc., are computed in [Wal01, ch. X]. These data as well as Waldspurger’s formula can be equally stated when $E/F$ is ramified. The main goal of this section is to show that Theorems 4.1, 4.10, 4.11 and 4.13 provide special cases and evidence that Waldspurger’s result could equally hold for ramified unitary groups, as well as evidence for Assem’s conjecture.

5.1 Stability

We state Assem’s stability conjecture in the unitary case.

Conjecture 5.1 [Ass98, Conjecture C, pp. 2].

Let $F$ be a non-archimedian local field with $\text{char}(F)=0$ or $\text{char}(F)\gg 0$ . For every stable nilpotent orbit ${\mathcal{O}}$ of a quasi-split unitary group $U_{n}(E/F)$ , there should be (up to constant) a unique linear combination of orbital integrals among the orbits in ${\mathcal{O}}$ that gives a stable distribution. All stable distributions supported on the nilpotent cone can be written as a linear combination of such stable distributions.

Waldspurger gave explicit formula for these combinations. If we restrict our attention to nilpotent orbits with (at most) two Jordan blocks, the formula of Waldspurger is simplified. For $0\leqslant m\leqslant n/2$ , denote by ${\mathcal{O}}_{m}(0)$ the set of nilpotent orbits with two Jordan blocks of sizes $n-m$ and $m$ (or regular nilpotent if $m=0$ ).

Theorem 5.2 (Waldspurger [Wal01, IX.15]).

Suppose $E/F$ is unramified, $\text{char}(k)>3n+1$ , and $\text{char}(F)=0$ . Then:

1. (a) for any $0\leqslant m\leqslant n/2$ with $2|mn$ ,

   $$\begin{eqnarray}\mathop{\sum }_{{\tilde{N}}_{m}\in {\mathcal{O}}_{m}(0)}J({\tilde{N}}_{m},\cdot )\text{ is a stable distribution;}\end{eqnarray}$$

2. (b) for any $0<m<n/2$ with $2\nmid mn$ , we have $\#{\mathcal{O}}_{m}(0)=2$ ; denote by ${\tilde{N}}_{m}^{(1)}$ and ${\tilde{N}}_{m}^{(2)}$ these two orbits, then

   $$\begin{eqnarray}J({\tilde{N}}_{m}^{(1)},\cdot )-J({\tilde{N}}_{m}^{(2)},\cdot )\text{ is a stable distribution.}\end{eqnarray}$$

What we can prove using Theorems 4.1 and 4.11 is the following result.

Theorem 5.3. Suppose instead $E/F$ is ramified. Under the assumption $\text{char}(k)\not =2$ and either $\text{char}(F)=0$ or $\text{char}(F)>n$ , we have:

1. (i) the same results as in Theorem 5.2 are true for $m\leqslant 2$ .

2. (ii) the same results as in Theorem 5.2 are true for all $m$ assuming Conjecture 5.1 of Assem.

Proof. We take $\ell$ an even integer so that ${\tilde{N}}$ is conjugate to $\unicode[STIX]{x1D70B}^{\ell }{\tilde{N}}$ for every nilpotent ${\tilde{N}}\in \tilde{\mathfrak{g}}(F)$ (see e.g. [Tsa15a, § 3.1]). Most of the time (e.g. when $\text{char}(F)=0$ ) $\ell =2$ works.

Let $\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ be any lift of $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ as in the introduction. The theorem of Shalika states that, for every function $f\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))$ , there exists $N_{0}$ such that $\forall N\geqslant N_{0}$ , if we write $f_{(N)}(X)=f(\unicode[STIX]{x1D70B}^{\ell N}X)$ , then

$$\begin{eqnarray}J^{\text{st}}(\tilde{T},f_{(N)})=\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0)}\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}^{\text{st}}(\tilde{T})J({\mathcal{O}},f_{(N)}).\end{eqnarray}$$

Nilpotent orbital integrals have the property (due to the symplectic structure on ${\mathcal{O}}$ ) that $J({\mathcal{O}},f_{(N)})=q^{(\ell N\dim {\mathcal{O}})/2}J({\mathcal{O}},f)$ . This allows us to rewrite

$$\begin{eqnarray}J^{\text{st}}(\unicode[STIX]{x1D70B}^{-\ell N}\tilde{T},f)=J^{\text{st}}(\tilde{T},f_{(N)})=\mathop{\sum }_{d=0}^{(\dim \tilde{G}-\text{rk}_{\bar{F}}\tilde{G})/2}\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0),\dim {\mathcal{O}}=2d}\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}^{\text{st}}(\tilde{T})q^{\ell dN}J({\mathcal{O}},f).\end{eqnarray}$$

Now let $f$ be any ‘stable’ function; $J^{\text{st}}(\tilde{X},f)=0$ for every $\tilde{X}\in \tilde{\mathfrak{g}}^{\text{rs}}(F)$ . The left-hand side by very definition vanishes. Interpolating with enough different $N$ , we see that for every $d$ ,

$$\begin{eqnarray}\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0),\dim {\mathcal{O}}=2d}\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}^{\text{st}}(\tilde{T})J({\mathcal{O}},f)=0.\end{eqnarray}$$

In other words

$$\begin{eqnarray}\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0),\dim {\mathcal{O}}=2d}\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}^{\text{st}}(\tilde{T})J({\mathcal{O}},\cdot )\text{ is a stable distribution.}\end{eqnarray}$$

When $d=(\dim \tilde{G}-\text{rk}_{\bar{F}}\tilde{G})/2-m$ with $m\leqslant 2$ , the only nilpotent orbits with dimension $2d$ are those nilpotent orbits with two Jordan blocks of sizes $n-m$ and $m$ (or one with size $n$ if $m=0$ ). To use previous results on Shalika germs, we need the following lemma.

Lemma 5.4. For any $0\leqslant m\leqslant n/2$ , there exists $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ such that $a_{m}(T)\not =0$ .

Proof. We use a parity trick. For each $0\leqslant m<n/2$ , we claim the existence of some $T$ for which $a_{m}(T)$ is odd. When $m=g+1$ and $n=2g+2$ , we observe that $a_{g+1}(T)/(q+1)$ has the same parity as $a_{g}(T)/(q+1)$ and thus we reduce to the case $m=g$ .

The idea is that the hyperelliptic involution gives an involution on $\operatorname{Sym}^{m}(C_{T})(k)$ . The parity of $\#\operatorname{Sym}^{m}(C_{T})(k)$ is thus given by the number of fixed points that are defined over $k$ , which in terms depends on the Galois structure on the Weierstrass points, or equivalently, how the characteristic polynomial factors in $k[x]$ .

Using Lemma 4.7(ii) and (iii) one can show the following: take $T$ so that $p_{T}(x)$ is an irreducible separable monic degree $n$ polynomial. Take $T^{\prime }$ so that $p_{T^{\prime }}(x)$ is another separable monic polynomial with two irreducible factors of degree $m$ and $n-m$ . Then $a_{m}(T)\not \equiv a_{m}(T^{\prime })$ (mod  $2$ ).◻

Theorems 4.1 and 4.11 give us $\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}^{\text{st}}(\tilde{T})=a_{m}(T)$ , or $-a_{m}(T)$ for one of the orbits if both $m$ and $n$ are odd. This completes part (i) of the theorem. For part (ii), simply note that Conjecture 5.1 allows us to separate nilpotent orbits with two Jordan blocks out (or nilpotent orbits of any type of Jordan blocks) for stability question.◻

5.2 Endoscopic transfer

The flow of this subsection is parallel to the previous section. However, we will encounter interesting geometric and combinatorial identities that can be thought as consequences of endoscopy. Recall that our endoscopy group of $\tilde{G}=U_{n}(E/F)$ is $U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)$ with $n_{1}+n_{2}=n$ . We write $\tilde{\mathfrak{g}}_{1}=\text{Lie }U_{n_{1}}(E/F)$ and $\tilde{\mathfrak{g}}_{2}=\text{Lie }U_{n_{2}}(E/F)$ .

We begin by stating the corresponding conjecture of Assem. The original conjecture of Assem for endoscopic transfer of nilpotent orbits comes from an induction construction due to Lusztig and Spaltenstein [LS79] and makes use of the Springer correspondence (see [Ass98, 4.3]). In our case $U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)$ is isomorphic to a twisted Levi subgroup of $U_{n}(E/F)$ , in which case this description has a simpler description in [LS79, § 1].

We summary their construction: let $\boldsymbol{G}$ be a reductive group over an algebraically closed field $\bar{F}$ and $\boldsymbol{M}$ a Levi subgroup. Take $\boldsymbol{P}=\boldsymbol{M}\boldsymbol{N}\subset \boldsymbol{G}$ any parabolic subgroup for the Levi, where $\boldsymbol{N}$ is its unipotent radical. For any nilpotent orbit ${\mathcal{O}}$ of $\text{Lie }\boldsymbol{M}$ , the variety ${\mathcal{O}}\cdot \text{Lie }\boldsymbol{N}$ has a dense open subset contained in some nilpotent orbit ${\mathcal{O}}^{\prime }$ of $\boldsymbol{G}$ . We then denote $\text{ind}_{\boldsymbol{M}}^{\boldsymbol{G}}{\mathcal{O}}:={\mathcal{O}}^{\prime }$ . In general when the reductive groups are defined over $F$ , this should be understood as an induction between stable orbits.

In our case, $\boldsymbol{G}=U_{n}(E/F)$ and $\boldsymbol{M}=U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)$ with $n_{1}+n_{2}=n$ . The induction for nilpotent orbits with two Jordan blocks is especially clear: if ${\tilde{N}}_{m_{1}}^{1}$ and ${\tilde{N}}_{m_{2}}^{2}$ are stable nilpotent orbits in $\tilde{\mathfrak{g}}_{1}$ (respectively $\tilde{\mathfrak{g}}_{2}$ ) with two Jordan blocks of sizes $(n_{1}-m_{1},m_{1})$ and $(n_{2}-m_{2},m_{2})$ where $2m_{i}\leqslant n_{i}$ , then $\text{ind}_{U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)}^{U_{n}(E/F)}{\tilde{N}}_{m_{1}}^{1}\times {\tilde{N}}_{m_{2}}^{2}={\tilde{N}}_{m}$ , the stable nilpotent orbits with two Jordan blocks of sizes $(n-m,m)$ , with $m=m_{1}+m_{2}$ .

We now assume Conjecture 5.1. Consider any stable combination ${\mathcal{D}}_{1}$ of nilpotent orbital integrals of $U_{n_{1}}(E/F)$ on orbits in a stable nilpotent orbit ${\mathcal{O}}_{1}\subset \tilde{\mathfrak{g}}_{1}(F)$ , and likewise another stable combination ${\mathcal{D}}_{2}$ of $U_{n_{2}}(E/F)$ on orbits in a stable nilpotent orbit ${\mathcal{O}}_{2}\subset \tilde{\mathfrak{g}}_{2}(F)$ . They give a stable nilpotent distribution ${\mathcal{D}}_{1}\otimes {\mathcal{D}}_{2}$ on $\tilde{\mathfrak{g}}_{1}(F)\times \tilde{\mathfrak{g}}_{2}(F)$ by ${\mathcal{D}}_{1}\otimes {\mathcal{D}}_{2}(f_{1}\otimes f_{2})={\mathcal{D}}_{1}(f_{1}){\mathcal{D}}_{2}(f_{2})$ .

Conjecture 5.5 [Ass98, Conjecture D, pp. 83].

The endoscopic transfer of ${\mathcal{D}}_{1}\otimes {\mathcal{D}}_{2}$ given above is a linear combination of nilpotent orbital integrals on orbits which lie in the stable orbit $\text{ind}\;\!_{U_{n_{1}}(E/F)\times U_{n_{2}}(E/F)}^{U_{n}(E/F)}({\mathcal{O}}_{1}\times {\mathcal{O}}_{2})$ .

As in Theorem 5.2, Waldspurger proved the conjecture in the case of unramified classical groups. We formulate some cases of Waldspurger’s result with two Jordan blocks. Let ${\mathcal{O}}_{m_{1}}^{1}(0)$ be the set of nilpotent orbits of $U_{n_{1}}(E/F)$ with two Jordan blocks of sizes $n_{1}-m_{1}$ and $m_{1}$ . Similarly ${\mathcal{O}}_{m_{2}}^{2}(0)$ and ${\mathcal{O}}_{m}(0)$ are used for nilpotent orbits of $U_{n_{2}}(E/F)$ and $U_{n}(E/F)$ . We have the following result.

Theorem 5.6 (Waldspurger [Wal01, XII.9]).

Suppose $E/F$ is unramified and $\text{char}(k)>3n+1$ .

1. (a) Suppose $n_{1}$ is odd and $n_{2}$ is even, so that $n=n_{1}+n_{2}$ is odd. Fix $0\leqslant 2m_{1}<n_{1}$ , $0\leqslant 2m_{2}\leqslant n_{2}$ and write $m=m_{1}+m_{2}$ . Write $\unicode[STIX]{x1D716}\in F^{\times }/N_{E/F}E^{\times }$ for the non-trivial class. For nilpotent orbit ${\tilde{N}}_{m}\in {\mathcal{O}}_{m}(0)$ , put $\unicode[STIX]{x1D6FE}({\tilde{N}}_{m})=-1$ if $m$ is odd and ${\tilde{N}}_{m}$ is the orbit classified by $((n-m,m),\unicode[STIX]{x1D716}\unicode[STIX]{x1D70B}^{-1/2},(-1)^{g})$ . In all other cases put $\unicode[STIX]{x1D6FE}({\tilde{N}}_{m})=1$ . We define likewise the factor $\unicode[STIX]{x1D6FE}({\tilde{N}}_{m_{1}})$ for ${\tilde{N}}_{m_{1}}\in {\mathcal{O}}_{m_{1}}^{1}(0)$ . Then

   $$\begin{eqnarray}\mathop{\sum }_{{\tilde{N}}_{m}\in {\mathcal{O}}_{m}(0)}\unicode[STIX]{x1D6FE}({\tilde{N}}_{m})^{m_{1}}J({\tilde{N}}_{m},\cdot )\end{eqnarray}$$
   is the endoscopic transfer of the stable distribution
   $$\begin{eqnarray}\mathop{\sum }_{{\tilde{N}}_{m_{1}}\in {\mathcal{O}}_{m_{1}}^{1}(0)}\unicode[STIX]{x1D6FE}({\tilde{N}}_{m_{1}})\mathop{\sum }_{{\tilde{N}}_{m_{2}}\in {\mathcal{O}}_{m_{2}}^{2}(0)}J({\tilde{N}}_{m_{1}},\cdot )\otimes J({\tilde{N}}_{m_{2}},\cdot ).\end{eqnarray}$$

2. (b) Suppose both $n_{1}$ and $n_{2}$ are even, so that $n=n_{1}+n_{2}$ is also even. Fix $0\leqslant 2m_{1}\leqslant n_{1}$ , $0\leqslant 2m_{2}\leqslant n_{2}$ and write $m=m_{1}+m_{2}$ . For any ${\tilde{N}}_{m}\in {\mathcal{O}}_{m}(0)$ , put $\unicode[STIX]{x1D6FE}({\tilde{N}}_{m})=1$ if ${\tilde{N}}_{m}$ is hyperbolic (see Definition 4.12) and $\unicode[STIX]{x1D6FE}({\tilde{N}}_{m})=-1$ if ${\tilde{N}}_{m}$ is elliptic. Then

   $$\begin{eqnarray}\mathop{\sum }_{{\tilde{N}}_{m}\in {\mathcal{O}}_{m}(0)}\unicode[STIX]{x1D6FE}({\tilde{N}}_{m})^{m_{1}}J({\tilde{N}}_{m},\cdot )\end{eqnarray}$$
   is the endoscopic transfer of the stable distribution
   $$\begin{eqnarray}\mathop{\sum }_{{\tilde{N}}_{m_{1}}\in {\mathcal{O}}_{m_{1}}^{1}(0)}\mathop{\sum }_{{\tilde{N}}_{m_{2}}\in {\mathcal{O}}_{m_{2}}^{2}(0)}J({\tilde{N}}_{m_{1}},\cdot )\otimes J({\tilde{N}}_{m_{2}},\cdot ).\end{eqnarray}$$

Parallel to Theorem 5.3, what we can show using Theorems 4.1, 4.10, 4.11 and 4.13 is the following result.

Theorem 5.7. Suppose instead $E/F$ is ramified. Then we have:

1. (i) the same results in Theorem 5.6 are true for $m=m_{1}+m_{2}\leqslant 2$ ;

2. (ii) the same results in Theorem 5.6 are true for all $m_{1}$ , $m_{2}$ assuming Conjectures 5.1 and 5.5.

Proof. We only give the proof for case (a). The proof for case (b) is completely the same while replacing the role of Theorems 4.1 and 4.10 by Theorems 4.11 and 4.13. The reason that in case (b) we want to assume both $n_{1}$ and $n_{2}$ are even (instead of only $n=n_{1}+n_{2}$ is even) is that in Theorem 4.13 we are only able to compute $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})+\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\text{ad}(u)\tilde{T})$ . It happens that this discrepancy matters exactly when $n_{1}$ is odd.

The idea is similar to the proof of Theorem 5.3. Let $\mathbf{x}_{1}$ be a vertex on the Bruhat–Tits building of $U_{n_{1}}(E/F)$ with reductive quotient $\text{SO}_{n_{1}}/_{k}$ . Let $V_{1}$ be the quasi-split quadratic space which is the standard representation of this $\text{SO}_{n_{1}}$ . Let $T_{1}$ be any regular semisimple self-adjoint endomorphism of $V_{1}$ . We have the same notation for $U_{n_{2}}(E/F)$ and let $T_{2}$ be any regular semisimple self-adjoint endomorphism of $V_{2}$ .

Let $p_{T_{1}}(x),p_{T_{2}}(x)\in k[x]$ denote respectively the monic characteristic polynomials of $T_{1}$ and $T_{2}$ . We assume that $p_{T_{1}}(x)$ and $p_{T_{2}}(x)$ are coprime. Write $C_{T_{1}}=(y^{2}=p_{T_{1}}(x))$ , $C_{T_{2}}=(y^{2}=p_{T_{2}}(x))$ , $J_{T_{1}}=\operatorname{Pic}^{0}(C_{T_{1}})$ and $J_{T_{2}}=\operatorname{Pic}^{0}(C_{T_{2}})$ . By abuse of notation (as we do not have $T$ yet), we also write $p_{T}(x)=p_{T_{1}}(x)p_{T_{2}}(x)$ a degree $n$ monic polynomial, $C_{T}=(y^{2}=p_{T}(x))$ and $J_{T}=\operatorname{Pic}^{0}(C_{T})$ .

The $G(0)(k)$ -orbit of actual such $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ with characteristic polynomial $p_{T}(x)$ is a torsor under $H^{1}(k,J_{T}[2])$ . This torsor is in fact canonically trivial [BG14, Proposition 4] as mentioned in the introduction; the identity element in $H^{1}(k,J_{T}[2])$ corresponds to the $T$ for which $(\times 2)^{-1}(\infty )\subset F_{T}$ is a trivial $J_{T}[2]$ -torsor (Theorem 3.1, [BG14, Proposition 4] and [Wan13, Corollary 2.5 and Proposition 2.29]). This orbit of $T$ is also the one that intersects with the Kostant section [BG13, § 7]. From now on we will use the symbol $T$ to denote a representative of this orbit for which $(\times 2)^{-1}(\infty )$ is trivial.

Let $\tilde{T}_{1}\in \tilde{\mathfrak{g}}_{1}(F)_{\mathbf{x}_{1},-1/2}$ be a lift of $T_{1}$ and likewise for $\tilde{T}_{2}$ . The orbits of those $\tilde{T}\in \tilde{\mathfrak{g}}(F)$ that ‘matches’ with $(\tilde{T}_{1},\tilde{T}_{2})\in \tilde{\mathfrak{g}}_{1}(F)\times \tilde{\mathfrak{g}}_{2}(F)$ , i.e. that has the same characteristic polynomial, enjoy a one-to-one correspondence with those orbits of $T$ classified by $H^{1}(k,J_{T}[2])$ in the last paragraph, thanks to Lemma 4.5.

Recall that $J_{T}[2]\cong \text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}/\unicode[STIX]{x1D707}_{2}\cong \ker (\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . In the middle group the $\unicode[STIX]{x1D707}_{2}$ is embedded into $\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}$ via the diagonal embedding. The second group and the third group are also dual to each other; this gives a self-dual structure $J_{T}[2]\times J_{T}[2]\rightarrow \unicode[STIX]{x1D707}_{2}$ .

Now as $p_{T}(x)=p_{T_{1}}(x)p_{T_{2}}(x)$ , we have $\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}=\text{Res}_{k}^{k[x]/p_{T_{1}}(x)}\unicode[STIX]{x1D707}_{2}\times \text{Res}_{k}^{k[x]/p_{T_{2}}(x)}\unicode[STIX]{x1D707}_{2}$ . On the latter group that is an element $\unicode[STIX]{x1D705}=\unicode[STIX]{x1D705}_{n_{1},n_{2}}:=(1,-1)$ . Since $\deg p_{T_{2}}=n_{2}$ is even, this element lies in $H^{0}(k,\ker (\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2}))\cong H^{0}(k,J_{T}[2])\cong H^{1}(k,J_{T}[2]^{\ast })^{\ast }\cong H^{1}(k,J_{T}[2])^{\ast }$ . In other words, $\unicode[STIX]{x1D705}$ defines a character on $H^{1}(k,J_{T}[2])$ .

By carefully checking the transfer factor, one can conclude that

$$\begin{eqnarray}\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})J(\tilde{T}_{\unicode[STIX]{x1D6FC}},\cdot )\end{eqnarray}$$

is the endoscopy transfer of

$$\begin{eqnarray}J^{\text{st}}(\tilde{T}_{1},\cdot )\otimes J^{\text{st}}(\tilde{T}_{2},\cdot )=\mathop{\sum }_{\unicode[STIX]{x1D6FC}_{1}\in H^{1}(k,J_{T_{1}}[2])}\mathop{\sum }_{\unicode[STIX]{x1D6FC}_{2}\in H^{1}(k,J_{T_{2}}[2])}J(\tilde{T}_{\unicode[STIX]{x1D6FC}_{1}},\cdot )\otimes J(\tilde{T}_{\unicode[STIX]{x1D6FC}_{2}},\cdot ).\end{eqnarray}$$

Here $\tilde{T}_{\unicode[STIX]{x1D6FC}}\in \tilde{\mathfrak{g}}(F)$ is any representative of the orbit classified by $\unicode[STIX]{x1D6FC}$ as described, and similarly for $\tilde{T}_{\unicode[STIX]{x1D6FC}_{1}}\in \tilde{\mathfrak{g}}_{1}(F)$ , $\tilde{T}_{\unicode[STIX]{x1D6FC}_{2}}\in \tilde{\mathfrak{g}}_{2}(F)$ . Arguing as in the proof of Theorem 5.3 and assume Conjectures 5.1 and 5.5 if $m>2$ , we have

$$\begin{eqnarray}\mathop{\sum }_{{\tilde{N}}_{m}\in {\mathcal{O}}_{m}(0)}\biggl(\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T}_{\unicode[STIX]{x1D6FC}})\biggr)J({\tilde{N}}_{m},\cdot )\end{eqnarray}$$

is the endoscopy transfer of

$$\begin{eqnarray}\mathop{\sum }_{\substack{ m_{1}+m_{2}=m \\ 0\leqslant 2m_{1}<n_{1} \\ 0\leqslant 2m_{2}\leqslant n_{2}}}~\mathop{\sum }_{{\tilde{N}}_{m_{1}}\in {\mathcal{O}}_{m_{1}}^{1}(0)}~\mathop{\sum }_{{\tilde{N}}_{m_{2}}\in {\mathcal{O}}_{m_{2}}^{2}(0)}\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m_{1}}}^{\text{st}}(\tilde{T}_{1})\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m_{2}}}^{\text{st}}(\tilde{T}_{2})\cdot J({\tilde{N}}_{m_{1}},\cdot )\otimes J({\tilde{N}}_{m_{2}},\cdot ).\end{eqnarray}$$

Later we will simply write $m_{1}+m_{2}=m$ for the first summation in the last formula while it should be understood that $m_{1}$ and $m_{2}$ vary only in the range for which ${\tilde{N}}_{m_{1}}$ and ${\tilde{N}}_{m_{2}}$ are defined. The key is to prove the following result.

Proposition 5.8. We have equality

$$\begin{eqnarray}\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T}_{\unicode[STIX]{x1D6FC}})=\mathop{\sum }_{m_{1}+m_{2}=m}\unicode[STIX]{x1D6FE}({\tilde{N}}_{m_{1}})\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m_{1}}}^{\text{st}}(\tilde{T}_{1})\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m_{2}}}^{\text{st}}(\tilde{T}_{2}),\end{eqnarray}$$

where in the summation in the right-hand side ${\tilde{N}}_{m_{1}}$ is chosen arbitrarily in ${\mathcal{O}}_{m_{1}}^{1}(0)$ and ${\tilde{N}}_{m_{2}}$ is chosen arbitrarily in ${\mathcal{O}}_{m_{2}}^{2}(0)$ . See the definition of $\unicode[STIX]{x1D6FE}(\cdot )$ in the statement of Theorem 5.6.

Proof. Using Theorems 4.1, 4.10 and 4.11, what we have to prove is the following geometric identity that underlies this endoscopic transfer:

(5.1) $$\begin{eqnarray}\displaystyle & & \displaystyle \displaystyle \mathop{\sum }_{m_{1}+m_{2}=m}a_{m_{1}}(T_{1})a_{m_{2}}(T_{2})\nonumber\\ \displaystyle & & \displaystyle \quad =\frac{1}{\#J_{T}[2](k)}\cdot \displaystyle \bigg(\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})\bigg(\mathop{\sum }_{0\leqslant 2\ell \leqslant m}\#\widetilde{\operatorname{Sym}}^{m-2\ell }(C_{T_{\unicode[STIX]{x1D6FC}}})(k)\cdot (-q)^{\ell }\cdot C_{\ell }(-g+m-2\ell -1)\nonumber\\ \displaystyle & & \displaystyle \qquad \displaystyle -\,(q+1)\mathop{\sum }_{0<2\ell +1\leqslant m}\#\widetilde{\operatorname{Sym}}^{m-2\ell -1}(C_{T_{\unicode[STIX]{x1D6FC}}})(k)\cdot (-q)^{\ell }\cdot C_{\ell }(-g+m-2\ell )\!\bigg)\!\bigg).\end{eqnarray}$$

We have to explain the slight abuse of notation here. Different $\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])$ gives us the same $C_{T_{\unicode[STIX]{x1D6FC}}}=C_{T}$ . However, the definition of the étale $J_{T}[2]$ -cover $\widetilde{\operatorname{Sym}}^{m}(C_{T_{\unicode[STIX]{x1D6FC}}})$ of $\operatorname{Sym}^{m}(C_{T})$ depends on the orbit of $T_{\unicode[STIX]{x1D6FC}}$ , thus depends on $\unicode[STIX]{x1D6FC}$ . In fact, changing $\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])$ exactly amounts to changing the Frobenius structure on $\widetilde{\operatorname{Sym}}^{m}(C_{T_{\unicode[STIX]{x1D6FC}}})$ as a $J_{T}[2]$ -torsor over $\operatorname{Sym}^{m}(C_{T})$ .

Recall that $T$ is just the $T_{\unicode[STIX]{x1D6FC}}$ with $\unicode[STIX]{x1D6FC}$ trivial. For any $\unicode[STIX]{x1D705}^{\prime }\in H^{1}(k,J_{T}[2])^{\ast }=H^{0}(k,J_{T}[2]^{\ast })$ , we can consider the $\unicode[STIX]{x1D705}^{\prime }$ -isotypic component $H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T_{\unicode[STIX]{x1D6FC}}}))_{\unicode[STIX]{x1D705}^{\prime }}$ . We have

$$\begin{eqnarray}\text{Tr}(\text{Frob}\,:\,H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T_{\unicode[STIX]{x1D6FC}}})/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}^{\prime }})=\unicode[STIX]{x1D705}^{\prime }(\unicode[STIX]{x1D6FC})\cdot \text{Tr}(\text{Frob}\,:\,H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T})/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}^{\prime }}).\end{eqnarray}$$

Summing over all $\unicode[STIX]{x1D6FC}$ and all $\unicode[STIX]{x1D705}^{\prime }$ , we see

(5.2) $$\begin{eqnarray}\displaystyle & & \displaystyle \mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})\#\widetilde{\operatorname{Sym}}^{m}(C_{T_{\unicode[STIX]{x1D6FC}}})(k)\nonumber\\ \displaystyle & & \displaystyle \quad =(-1)^{m}\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})\mathop{\sum }_{\unicode[STIX]{x1D705}^{\prime }\in H^{1}(k,J_{T}[2])\ast }\unicode[STIX]{x1D705}^{\prime }(\unicode[STIX]{x1D6FC})\cdot \text{Tr}(\text{Frob}\,:\,H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T})/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}^{\prime }}).\nonumber\\ \displaystyle & & \displaystyle \quad =(-1)^{m}\#J_{T}[2](k)\cdot \text{Tr}(\text{Frob}\,:\,H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T}))_{\unicode[STIX]{x1D705}}/_{\bar{k}},\mathbb{Q}_{\ell }).\end{eqnarray}$$

In the last step we used the equality $\#H^{1}(k,J_{T}[2])=\#J_{T}[2](k)$ . To compute $\text{Tr}(\text{Frob}\,:\,H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T})/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}}\!)$ , it will be a good idea to first deal with the case $m=1$ . In the rest of the proof we write $\tilde{C}_{T}:=\widetilde{\operatorname{Sym}}^{1}(C_{T})$ . This is an étale $J_{T}[2]$ -cover of $C_{T}$ .

Finite covers between (projective smooth) curves can be read out from their function fields. Let us base change from the ground field $k$ to $\bar{k}$ for the moment. Recall $C_{T}$ is a double cover of $\mathbb{P}^{1}$ . Their function fields are respectively $\bar{k}(x)\subset \bar{k}(x,\sqrt{p_{T}(x)})$ . The key is to observe

$$\begin{eqnarray}\bar{k}(\tilde{C}_{T})=\bar{k}(x,\sqrt{p_{T}(x)},\sqrt{P(x)}\mid P(x)\text{ runs over even degree divisors of }p_{T}(x)).\end{eqnarray}$$

This is because the above function field extension gives an étale $J_{T}[2]$ -cover of $C_{T}$ , which is unique over $\bar{k}$ . Now $\unicode[STIX]{x1D705}$ , being a non-trivial element in $J_{T}[2](k)^{\ast }$ , corresponds to a degree two cover $C_{T}^{\unicode[STIX]{x1D705}}$ of $C_{T}$ inside $\tilde{C}_{T}\rightarrow C_{T}$ . This cover is given by the function field $\bar{k}(C_{T}^{\unicode[STIX]{x1D705}})=\bar{k}(x,\sqrt{p_{T}(x)},\sqrt{p_{T_{2}}(x)})=\bar{k}(x,\sqrt{p_{T_{1}}(x)},\sqrt{p_{T_{2}}(x)})$ .

The curve $C_{T}^{\unicode[STIX]{x1D705}}$ , as well as its function field, descend back to $k$ . Precisely, since $\tilde{C}_{T}$ is defined to be the curve for which the fiber above $\infty$ is trivial, we have $C_{T}^{\unicode[STIX]{x1D705}}=k(x,\sqrt{p_{T}(x)},\sqrt{p_{T_{2}}(x)})$ (here it is important that $p_{T_{2}}(x)$ was chosen to be monic). Now recall the following lemma.

Lemma 5.9. Let $X$ be a quasi-projective variety over any field $k$ and $G$ be a finite group acting on $X$ . Choose prime $\ell$ which is coprime to $|G|$ . Let $Y=X/G$ be the scheme-theoretic quotient. Then $H^{\ast }(Y/_{\bar{k}},\mathbb{Q}_{\ell })\cong H^{\ast }(X/_{\bar{k}},\mathbb{Q}_{\ell })^{G}$ .

Using the lemma, we have

$$\begin{eqnarray}H^{\ast }(\tilde{C}_{T}/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}}=H^{\ast }(C_{T}^{\unicode[STIX]{x1D705}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell }),\end{eqnarray}$$

where the two sides of the equality are in the abelian category of virtual representations of the free abelian group generated by Frobenius. Nevertheless, it is obvious from the function field of $C_{T}^{\unicode[STIX]{x1D705}}$ that it is a $(\unicode[STIX]{x1D707}_{2})^{2}$ -cover of $\mathbb{P}^{1}$ , and that the three double covers in the middle are $C_{T}$ , $C_{T_{1}}$ and $C_{T_{2}}$ . This gives

$$\begin{eqnarray}\displaystyle & & \displaystyle H^{\ast }(C_{T}^{\unicode[STIX]{x1D705}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }(C_{T}/_{\bar{k}},\mathbb{Q}_{\ell })\nonumber\\ \displaystyle & & \displaystyle \quad =(H^{\ast }(C_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }(\mathbb{P}^{1}/_{\bar{k}},\mathbb{Q}_{\ell }))\oplus (H^{\ast }(C_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }(\mathbb{P}^{1}/_{\bar{k}},\mathbb{Q}_{\ell }))\nonumber\\ \displaystyle & & \displaystyle \quad =H^{1}(C_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })\oplus H^{1}(C_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell }).\nonumber\end{eqnarray}$$

In summary $H^{\ast }(\tilde{C}_{T}/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}}=H^{1}(C_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })\oplus H^{1}(C_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell })$ . For general $m$ , what we have is the following lemma.

Lemma 5.10. Let $J_{T_{1}}$ and $J_{T_{2}}$ be the Jacobian of $C_{T_{1}}$ and $C_{T_{2}}$ , respectively. Then

(5.3) $$\begin{eqnarray}H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T})/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}}=\bigoplus _{d=0}^{m}H^{d}(J_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })\otimes H^{m-d}(J_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell }).\end{eqnarray}$$

To prove Lemma 5.10, note that $\unicode[STIX]{x1D70B}_{1}^{\acute{e} t,tame}(\operatorname{Sym}^{m}(C_{T}))\cong \unicode[STIX]{x1D70B}_{1}^{\acute{e} t,tame}(C_{T})$ canonically, and thus we have a double cover $\operatorname{Sym}^{m}(C_{T})^{\unicode[STIX]{x1D705}}\rightarrow \operatorname{Sym}^{m}(C_{T})$ corresponding to $C_{T}^{\unicode[STIX]{x1D705}}\rightarrow C_{T}$ . This double cover can be seen as a $S_{m}$ -quotient of $((C_{T})^{m})^{\unicode[STIX]{x1D705}}\rightarrow (C_{T})^{m}$ , the ‘diagonal’ double cover in the $(\unicode[STIX]{x1D707}_{2})^{m}$ -cover $(C_{T}^{\unicode[STIX]{x1D705}})^{m}\rightarrow (C_{T})^{m}$ .

Now the cover $((C_{T})^{m})^{\unicode[STIX]{x1D705}}$ is a $((\unicode[STIX]{x1D707}_{2})^{m}\times \unicode[STIX]{x1D707}_{2})$ -cover of $(\mathbb{P}^{1})^{m}$ . For any $\unicode[STIX]{x1D708}\in \{1,2\}^{m}$ , denote by $V^{\unicode[STIX]{x1D708}}$ the ‘diagonal’ double cover of $(\mathbb{P}^{1})^{m}$ in $\prod _{i=1}^{m}C_{T_{\unicode[STIX]{x1D708}(i)}}\rightarrow (\mathbb{P}^{1})^{m}$ . These are exactly all of the double covers of $(\mathbb{P}^{1})^{m}$ which are between $((C_{T})^{m})^{\unicode[STIX]{x1D705}}\rightarrow (\mathbb{P}^{1})^{m}$ but not between $(C_{T})^{m}\rightarrow (\mathbb{P}^{1})^{m}$ . We thus have

$$\begin{eqnarray}H^{\ast }(((C_{T})^{m})^{\unicode[STIX]{x1D705}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }((C_{T})^{m}/_{\bar{k}},\mathbb{Q}_{\ell })=\mathop{\sum }_{\unicode[STIX]{x1D708}\in \{1,2\}^{m}}(H^{\ast }(V^{\unicode[STIX]{x1D708}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }((\mathbb{P}^{1})^{m}/_{\bar{k}},\mathbb{Q}_{\ell })).\end{eqnarray}$$

On the other hand, for the $(\unicode[STIX]{x1D707}_{2})^{m}$ -cover $\prod _{i=1}^{m}C_{T_{\unicode[STIX]{x1D708}(i)}}\rightarrow (\mathbb{P}^{1})^{m}$ , we can consider the product map $\unicode[STIX]{x1D719}\,:\,\unicode[STIX]{x1D707}_{2}^{m}\rightarrow \unicode[STIX]{x1D707}_{2}$ . Then the $\unicode[STIX]{x1D719}$ -isotypic part is equal to the term in the previous sum:

$$\begin{eqnarray}H^{\ast }\mathop{\biggl(\mathop{\prod }_{i=1}^{m}C_{T_{\unicode[STIX]{x1D708}(i)}}/_{\bar{k}},\mathbb{Q}_{\ell }\biggr)}\nolimits_{\unicode[STIX]{x1D719}}\cong H^{\ast }(V^{\unicode[STIX]{x1D708}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }((\mathbb{P}^{1})^{m}/_{\bar{k}},\mathbb{Q}_{\ell }),\quad \forall \unicode[STIX]{x1D708}\in \{1,2\}^{m}.\end{eqnarray}$$

Nevertheless, Künneth formula implies $H^{\ast }(\prod _{i=1}^{m}C_{T_{\unicode[STIX]{x1D708}(i)}}/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D719}}=\bigotimes _{i=1}^{m}H^{1}(C_{T_{\unicode[STIX]{x1D708}(i)}}/_{\bar{k}},\mathbb{Q}_{\ell })$ . We thus have

$$\begin{eqnarray}H^{\ast }(((C_{T})^{m})^{\unicode[STIX]{x1D705}}/_{\bar{k}},\mathbb{Q}_{\ell })\ominus H^{\ast }((C_{T})^{m}/_{\bar{k}},\mathbb{Q}_{\ell })=\mathop{\sum }_{\unicode[STIX]{x1D708}\in \{1,2\}^{m}}\bigotimes _{i=1}^{m}H^{1}(C_{T_{\unicode[STIX]{x1D708}(i)}}/_{\bar{k}},\mathbb{Q}_{\ell }).\end{eqnarray}$$

Now the left-hand side of (5.3) is the $S_{m}$ -invariant part of the left-hand side above, taking $S_{m}$ -invariant on the right-hand side gives

$$\begin{eqnarray}H^{\ast }(\widetilde{\operatorname{Sym}}^{m}(C_{T})/_{\bar{k}},\mathbb{Q}_{\ell })_{\unicode[STIX]{x1D705}}=\mathop{\sum }_{d=0}^{m}\operatorname{Sym}^{d}H^{1}(C_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })\otimes \operatorname{Sym}^{m-d}H^{1}(C_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell }),\end{eqnarray}$$

where on the right-hand side, $d$ correspond to the number of $i$ with $\unicode[STIX]{x1D708}(i)=1$ . Here (!) the $\operatorname{Sym}^{d}H^{1}$ above has the meaning of the $d$ th symmetric power of (virtual) representations as super (i.e. ( $\mathbb{Z}/2\mathbb{Z}$ )-graded) vector spaces; that is, $\operatorname{Sym}^{d}H^{1}=\bigwedge ^{d}H^{1}$ in the usual notation. This proves (5.3).

Combining (5.2) and (5.3), we obtain

(5.4) $$\begin{eqnarray}\frac{1}{\#J_{T}[2](k)}\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in H^{1}(k,J_{T}[2])}\unicode[STIX]{x1D705}(\unicode[STIX]{x1D6FC})\#\widetilde{\operatorname{Sym}}^{m}(C_{T_{\unicode[STIX]{x1D6FC}}})(k)=\bigoplus _{d=0}^{m}H^{d}(J_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })\otimes H^{m-d}(J_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell }).\end{eqnarray}$$

It is now a matter of combinatorics to prove (5.1). First we have to rewrite $a_{m_{1}}(T)$ and $a_{m_{2}}(T)$ . In the odd case, that is for $a_{m_{1}}(T)$ , Lemma 4.7(iii) and Proposition B.6 together gives

$$\begin{eqnarray}a_{m_{1}}(T)=\mathop{\sum }_{0\leqslant 2\ell \leqslant m_{1}}q^{\ell }\cdot C_{\ell }(-g_{1}+m_{1}-2\ell )\cdot \text{Tr}(\text{Frob}\,:\,H^{m_{1}-2\ell }(J_{T_{1}}/_{\bar{k}},\mathbb{Q}_{\ell })),\end{eqnarray}$$

where $g_{1}$ is the genus of $C_{T_{1}}$ ; $n_{1}=2g_{1}+1$ . For the even case, the number $a_{m}(T)$ is like $a_{m}(T)-(q+1)a_{m-1}(T)$ if using the definition of the odd case. This gives

$$\begin{eqnarray}\displaystyle a_{m_{2}}(T) & = & \displaystyle \mathop{\sum }_{0\leqslant 2\ell \leqslant m_{2}}q^{\ell }\cdot C_{\ell }(-g_{1}+m_{2}-2\ell )\cdot \text{Tr}(\text{Frob}\,:\,H^{m_{2}-2\ell }(J_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell }))-(q+1)\nonumber\\ \displaystyle & & \displaystyle \times \mathop{\sum }_{0<2\ell +1\leqslant m_{2}}q^{\ell }\cdot C_{\ell }(-g_{1}+m_{2}-2\ell -1)\cdot \text{Tr}(\text{Frob}\,:\,H^{m_{2}-2\ell -1}(J_{T_{2}}/_{\bar{k}},\mathbb{Q}_{\ell })).\nonumber\end{eqnarray}$$

Having the expressions of $a_{m_{1}}(T)$ and $a_{m_{2}}(T)$ at hand, one sees that (5.1) follows from (5.4) and Corollary B.5. This finishes the proof of Proposition 5.8.◻

With the endoscopic transfer formula we had right before Proposition 5.8, it now suffices to show that when we run over all possible choices of coprime separable polynomials $p_{T_{1}}(x),p_{T_{2}}(x)\in k[x]$ of degree $n_{1}$ and $n_{2}$ , respectively, we have

$$\begin{eqnarray}\mathop{\sum }_{m_{1}+m_{2}=m}~\mathop{\sum }_{{\tilde{N}}_{m_{1}}\in {\mathcal{O}}_{m_{1}}^{1}(0)}~\mathop{\sum }_{{\tilde{N}}_{m_{2}}\in {\mathcal{O}}_{m_{2}}^{2}(0)}\unicode[STIX]{x1D6FE}({\tilde{N}}_{m_{1}})a_{m_{1}}(T_{1})a_{m_{2}}(T_{2})\cdot J({\tilde{N}}_{m_{1}},\cdot )\otimes J({\tilde{N}}_{m_{2}},\cdot )\end{eqnarray}$$

spans the linear space of stable distributions supported on the union of all ${\tilde{N}}_{m_{1}}\times {\tilde{N}}_{m_{2}}$ with $m_{1}+m_{2}=m$ .

In other words we have to prove the vectors $(a_{m_{1}}(T_{1})a_{m_{2}}(T_{2}))_{m_{1}+m_{2}=m}$ for different $T_{1},T_{2}$ span $\mathbb{Q}^{\{(m_{1},m_{2})\mid m_{1}+m_{2}=m,\;0\leqslant 2m_{1}<n_{1},\;0\leqslant 2m_{2}\leqslant n_{2}\}}$ . That this is always the case can be proved with a parity trick similar to Lemma 5.4. This finishes the proof of Theorem 5.7.◻

Remark 5.11. In fact, it was endoscopic transfer which led us into conjecturing the results in Theorems 4.1 and 4.11 before knowing how to compute them. The point is that without having a good method to compute Shalika germs, § 3 already tells us that the stable Shalika germs $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}(\tilde{T})$ should be expressed in terms of linear combinations of $\#\operatorname{Sym}^{m^{\prime }}(C_{T})(k)$ , $0\leqslant m^{\prime }\leqslant m$ . Together with Assem’s conjectures, this suggests that something such as (5.1), with some a priori unknown coefficients, should be true.

On the other hand, ${\tilde{N}}_{m}$ only exists as a nilpotent orbit with codimension $2m$ in the regular nilpotent orbit if $2m\leqslant n$ . In other words, this suggests that the stable Shalika germ formula (which we proved to be $a_{m}(T)$ ), should be something that vanishes when $2m>n$ . This together with some weaker computation was what led us to the formula $\unicode[STIX]{x1D6E4}_{{\tilde{N}}_{m}}^{\text{st}}(\tilde{T})=a_{m}(T)$ .

6 Local character expansions of supercuspidal representations

This section is devoted to the application of our Shalika germ formulas to local character expansion of specific supercuspidal representations. Briefly speaking, we use our result on supercuspidal representations whose local character looks like the Fourier transform of $J(\tilde{T},\cdot )$ to obtain a Harish-Chandra–Howe local character expansion, and invoke the interpretation of character expansion by Mœglin and Waldspurger [MW87]. For the use of the result in [MW87], we assume $\text{char}(F)=0$ in this section.

Again fix $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ . Recall $\mathfrak{g}(1)(k)\cong \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}\cong \tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},1}$ . Furthermore, we have that $\mathfrak{g}(1)(k)$ is self-dual. This allows us to identify $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}$ and $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},1}$ as the dual of each other. With a choice of non-trivial additive character $\unicode[STIX]{x1D713}\,:\,(k,+)\rightarrow \mathbb{C}^{\times }$ , the element $T$ give rises to a character on $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},1}\cong \tilde{G}(F)_{\mathbf{x},1/2}/\tilde{G}(F)_{\mathbf{x},1}$ , and thus a one-dimensional representation of $\tilde{G}(F)_{\mathbf{x},1/2}$ . We denote by $\unicode[STIX]{x1D713}_{T}$ this representation.

The compact induction

$$\begin{eqnarray}\unicode[STIX]{x1D70B}_{T}:=\text{c-ind}_{\tilde{G}(F)_{\mathbf{x},1/2}}^{\tilde{G}(F)}\unicode[STIX]{x1D713}_{T}=\{f\in C_{c}^{\infty }(\tilde{G}(F))\;|\;f(g_{1}g_{2})=\unicode[STIX]{x1D713}_{T}(g_{1})f(g_{2}),~\forall g_{1}\in \tilde{G}(F)_{\mathbf{x},1/2}\}.\end{eqnarray}$$

can be shown to be the direct sum of finitely many supercuspidal representations. Let $r$ be the number of irreducible factor of the characteristic polynomial $p_{T}(x)$ of $T$ and $L=k[x]/p_{T}(x)$ be an étale algebra over $k$ ; $L$ is the direct product of $r$ finite extensions of $k$ . We have $\text{Stab}_{O(V)}(T)=\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ has $2^{r}$ points defined over $k$ . Then $\unicode[STIX]{x1D70B}_{T}$ is the direct sum of $2^{r}$ distinct irreducible supercuspidal representations [RY14, Proposition 2.4] of depth $\frac{1}{2}$ . These are examples of epipelagic representations of Reeder and Yu [RY14].

Now let $\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}}$ be the character of $\unicode[STIX]{x1D70B}_{T}$ . In other words, $\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}}\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))$ is the ( $\tilde{G}(F)$ -conjugation) invariant distribution such that for any $f\in C_{c}^{\infty }(\tilde{G}(F))$ , $\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}}(f):=\text{Tr}(\unicode[STIX]{x1D70B}_{T}(f))$ . Here to define $\unicode[STIX]{x1D70B}_{T}(f)$ we need a choice of measure on $\tilde{G}(F)$ , which we give in Appendix A.

The basic philosophy that goes back to at least Harish-Chandra is that characters should be compared with Fourier transforms of orbital integrals. Use as in Appendix A the self-dual structure $\unicode[STIX]{x1D713}(B(\cdot ,\cdot ))$ and measure on $\tilde{\mathfrak{g}}(F)$ . This gives, for $f\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F))$ , its Fourier transform

$$\begin{eqnarray}\widehat{f}(\tilde{X}):=\int _{\tilde{\mathfrak{g}}(F)}\unicode[STIX]{x1D713}(B(\tilde{X},{\tilde{Y}}))\,d{\tilde{Y}}.\end{eqnarray}$$

We define $\widehat{J}(\tilde{X},f):=J(\tilde{X},\widehat{f})$ , the Fourier transform of orbital integrals. Fix a lift $\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}$ . What one has is the following lemma.

Lemma 6.1. We have:

1. (i) $\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}}$ is supported on $\tilde{G}(F)_{\mathbf{x},1/2}$ ;

2. (ii) let $\mathsf{e}\,:\,\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}\xrightarrow[]{{\sim}}\tilde{G}(F)_{\mathbf{x},1/2}$ be a mock exponential map (see [DeB02, Hypothesis 3.2.1], for us it can be given by the Cayley transform); then, for any $f\in C_{c}^{\infty }(\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2})$ ,

   $$\begin{eqnarray}\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}}(f\circ \mathsf{e})=2^{r}\cdot \widehat{J}(\tilde{T},f);\end{eqnarray}$$

3. (iii) for each of the $2^{r}$ components of $\unicode[STIX]{x1D70B}_{T}$ , its character (which has larger support), when restricted to $\tilde{G}(F)_{\mathbf{x},1/2}$ and pulled back to $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}$ via $\mathsf{e}$ , is equal to $\widehat{J}(\tilde{T},f)$ .

This is a very special case of the main result of [AS09], an interpretation of which is available at [Kal15, (5.1.1)]. The method in this special case works for $\text{char}(k)>2$ .

From now on let $\unicode[STIX]{x1D70B}_{T}^{o}$ be any fixed component of $\unicode[STIX]{x1D70B}_{T}$ , and $\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}^{o}}$ its character. Let ${\mathcal{O}}(0)$ be the set of nilpotent orbits. Then the Harish-Chandra–Howe local character expansion [HC99, Theorem 4] states that there exists constants $(c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o}))_{{\mathcal{O}}\in {\mathcal{O}}(0)}\in \mathbb{C}$ such that

(6.1) $$\begin{eqnarray}\unicode[STIX]{x1D6E9}_{\unicode[STIX]{x1D70B}_{T}^{o}}(f\circ \mathsf{e})=\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0)}c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})\widehat{J}({\mathcal{O}},f),\end{eqnarray}$$

for all $f$ that are supported in a sufficiently small neighborhood $U\subset \tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}$ of $0\in \tilde{\mathfrak{g}}(F)$ . On the other hand, in [MW87] Mœglin and Waldspurger proved that, if ${\mathcal{O}}^{\prime }$ is any nilpotent orbit satisfying that for any ${\mathcal{O}}$ whose boundary contains ${\mathcal{O}}^{\prime }$ we have $c_{{\mathcal{O}}}=0$ , then $c_{{\mathcal{O}}^{\prime }}(\unicode[STIX]{x1D70B}_{T}^{o})$ is equal to the dimension of the degenerated Whittaker model associated to ${\mathcal{O}}^{\prime }$ of $\unicode[STIX]{x1D70B}_{T}^{o}$ .

Now, restricting to the small neighborhood $U$ , we have by Lemma 6.1(iii) and (6.1) that

$$\begin{eqnarray}\widehat{J}(\tilde{T},f)=\mathop{\sum }_{{\mathcal{O}}\in {\mathcal{O}}(0)}c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})\widehat{J}({\mathcal{O}},f).\end{eqnarray}$$

By inversing the Fourier transform, we see the following result.

Corollary 6.2. We have $c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})=\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T})$ , the latter are given by formulas in Theorems 4.10 and 4.13.

Corollary 6.3. For any $m\geqslant 0$ we can find $C=C(m,q)$ such that for any $n\geqslant C$ , we can find supercuspidal representations of $U_{n}(E/F)$ of the form $\unicode[STIX]{x1D70B}_{T}^{o}$ such that $c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})=0$ for every nilpotent orbit ${\mathcal{O}}$ of two Jordan blocks of sizes $n-m^{\prime }$ and $m^{\prime }$ , $0\leqslant m^{\prime }\leqslant m$ . Here by abuse of language the $m^{\prime }=0$ case corresponds to an orbit with a single Jordan block, namely a regular nilpotent orbit.

Proof. Take $T$ so that $p_{T}(x)$ has as many irreducible factors as possible, so that $\#J_{T}[2](k)=2^{r}$ with $r>n/(1+\log _{q}n)$ . The varieties in Theorems 4.10 and 4.13 are $J_{T}[2]$ -covers of $\operatorname{Sym}^{m^{\prime }}(C_{T})$ (and $\operatorname{Sym}^{m^{\prime }}(C_{T}^{\prime })$ , etc.), whose numbers of points can be bounded by the Weil bound on the Frobenius trace. Now for any $J_{T}[2]$ -cover of $\operatorname{Sym}^{m^{\prime }}(C_{T})$ , the fiber above a rational point in $\operatorname{Sym}^{m^{\prime }}(C_{T})$ is a $J_{T}[2]$ -torsor. Recall that the orbits in the stable orbit of $T$ are classified by $H^{1}(k,J_{T}[2])$ , and when $T$ runs over all such orbits in the same stable orbit, the fiber above any chosen rational point in $\operatorname{Sym}^{m^{\prime }}(C_{T})$ will also run over all possible $J_{T}[2]$ -torsors.

Once $n$ is large enough, we have $r$ large enough so that $2^{r}$ will be much greater than $\sum _{m^{\prime }=0}^{m}\#\operatorname{Sym}^{m^{\prime }}(C_{T})(k)$ (and more for other covers and covers of $\operatorname{Sym}^{m^{\prime }}(C_{T}^{\prime })$ , etc.). We can thus find an orbit in the stable orbit of $T$ , i.e. a class in $H^{1}(k,J_{T}[2])$ , such that for the corresponding covers $\widetilde{\operatorname{Sym}}^{m^{\prime }}(C_{T})$ in Theorems 4.10 and 4.13, $0\leqslant m^{\prime }\leqslant m$ , the torsor above each rational point is non-trivial. That is to say $\widetilde{\operatorname{Sym}}^{m^{\prime }}(C_{T})$ (and similarly $\widetilde{\operatorname{Sym}}^{m^{\prime }}(C_{T}^{\prime }),\widetilde{\operatorname{Sym}}^{m^{\prime },\ast }(C_{T})$ , etc.) has no rational points. When $n$ is odd this says $c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})=\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T})=0$ , which is what we want. When $n$ is even we have instead $c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})+c_{\text{Ad}(u){\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})=\unicode[STIX]{x1D6E4}_{{\mathcal{O}}}(\tilde{T})+\unicode[STIX]{x1D6E4}_{\text{Ad}(u){\mathcal{O}}}(\tilde{T})=0$ . Since inductively by [MW87] we have $c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o}),c_{\text{Ad}(u){\mathcal{O}}}(\unicode[STIX]{x1D70B}_{T}^{o})\geqslant 0$ , we conclude that they all vanish.◻

Note a nilpotent with two Jordan blocks is never in the closure of a nilpotent orbit with more than two Jordan blocks. One can thus have many examples where the dimension of the degenerate Whittaker models are (up to constant) number of rational points on varieties in Theorems 4.10 and 4.13.

Example 6.4. For example, take $n=2g+1$ odd and take $p_{T}(x)\in k[x]$ any polynomial of degree $2g+1$ that is the product of $r$ distinct irreducible factors with $r>1$ . Let $C_{T}=(y^{2}=p_{T}(x))$ (the smooth completion). Take an étale Galois $J_{T}[2]$ -cover $\tilde{C}_{T}$ of $C_{T}$ for which the fiber above $\infty \in C_{T}$ is a non-trivial $J_{T}[2]$ -torsor. Such a choice corresponds to an orbit of such $T$ in its stable orbit. The corresponding representation has $c_{{\tilde{N}}_{0}}(\unicode[STIX]{x1D70B}_{T}^{o})=(1/\#J_{T}[2](k))\#\widetilde{\operatorname{Sym}}^{0}(C_{T})(k)=0$ and $c_{{\tilde{N}}_{1}}(\unicode[STIX]{x1D70B}_{T}^{o})=(1/\#J_{T}[2](k))\#\tilde{C}_{T}(k)$ , i.e. the dimension of the degenerate Whittaker model for the subregular orbit ${\tilde{N}}_{1}$ is $2^{-(r-1)}$ times the number of rational points on $\tilde{C}_{T}$ , a curve of genus $2^{2g}(g-1)+1$ over $k$ . It will be interesting to see how these points actually ‘live’ on the degenerate Whittaker model.