Double EPW sextics associated with Gushel–Mukai surfaces

Abstract

Works by O’Grady allow to associate with a two-dimensional Gushel–Mukai (GM) variety, which is a K3 surface, a double Eisenbud–Popescu–Walter (EPW) sextic. We characterize the $K3$ surfaces whose associated double EPW sextic is smooth. As a consequence, we are able to produce symplectic actions on some families of smooth double EPW sextics which are hyper-Kähler manifolds.

We also provide bounds for the automorphism group of GM varieties in dimension 2 and higher.

1. Introduction

Double Eisenbud–Popescu–Walter (EPW) sextics are an important family of hyper-Kähler manifolds, i.e. compact simply connected Kähler manifolds with a unique, up to scalar, holomorphic two-form, which is everywhere non-degenerate. The linebase example of hyper-Kähler manifolds are $K3$ surfaces; a classical example is double covers of $\mathbb{P}^2$. Double EPW sextics are a generalization of it, as they come with a structure of double covers of special sextic hypersurfaces in $\mathbb{P}^5$, the so-called EPW sextics.

In §2, we present the main objects that come into play. The basis of the theory of double EPW sextics has been developed by O’Grady in an influential series of articles. Already from [16], an important connection is observed between (double) EPW sextics and ordinary two-dimensional Gushel–Mukai (GM) varieties, complete intersections of a linear space and a quadric hypersurface inside a Grassmannian. This link has been successively extended to higher-dimensional GM varieties by Iliev and Manivel in [5] and then developed in detail by Debarre and Kuznetsov, in a series of articles that lay the groundwork for an extensive study of the beautiful and intricate interplay between the two families.

A natural question about this relation is: can we give conditions on a GM variety to be associated with a double EPW sextic which is smooth? Indeed, whenever the double cover is smooth, it is a hyper-Kähler manifold.

In §3, we answer this question for two-dimensional GM varieties, which are Brill–Noether general $\langle 10 \rangle$-polarized $K3$ surfaces. We provide an answer in terms of geometry and in terms of period: the following result summarizes theorems 3.2 and 3.3.

Theorem 1.1.

The double EPW sextic associated with a strongly smooth $K3$ surface $S=\mathbb{P}^6\cap G(2,V_5)\cap Q$ is smooth if and only if S contains neither lines nor quintic elliptic pencils. Equivalently, a $\langle 10 \rangle$-polarized $K3$ surface is associated with a double EPW sextic which is a hyper-Kähler manifold if and only if it does not lie in six (explicitly described) divisors in the corresponding moduli space.

This result relies on the tools provided by Debarre and Kuznetsov which we presented in §2 as well as on a careful description of the involution acting on the moduli space of EPW sextics, described for the first time by O’Grady [17]. It is interesting to note that smoothness was already known in the very general case, but theorem 1.1 holds without any general assumption: eliminating the hypothesis of very generality is usually a challenging problem and we believe that our result is interesting in this spirit.

As an application of theorem 1.1, in §4, we produce symplectic actions for various groups on families of double EPW sextics which are hyper-Kähler, from lattice-theoretic considerations on automorphisms of $K3$ surfaces.

Section 5 is devoted to finding bounds for the automorphism group of GM varieties and on actions of groups on EPW sextics. For example, in proposition 5.2, we show that the automorphism group of S is a finite subgroup of $PGL(2,\mathbb{C})$ and can only be symplectic; we also provide some results for GM varieties in higher dimension. Most of these results are obtained by studying the automorphism group of Fano varieties containing a GM variety.

A previous version of this article contained some results about lifting automorphisms of EPW sextics to their double cover; in the meanwhile, a much stronger result has been obtained by Kuznetsov in [29], see proposition 2.10. We used this result to simplify the proof of proposition 4.2 and to prove the symplecticity of the automorphism group of a strongly smooth $K3$ surface in $\mathbb{P}^6$, see proposition 5.2.

1.1. Notations

Let X be a topological space. A property holds for $x\in X$ general if the condition is satisfied by all the points inside an open subset of X. A property holds for $x\in X$ very general if the condition is satisfied by all the points in the complement of a countable union of closed subspaces inside X.

Given a complex vector space V, the Grassmannian of k-dimensional vector subspaces in V will be denoted by $G(k,V)$; every Grassmannian we consider will be embedded in the projective space $\mathbb{P}(\bigwedge^k V)$ via the Plücker embedding. Non-zero decomposable vectors in $\mathbb{P}(\bigwedge^k V)$ are elements that lie in $G(k,V)$.

Definition 1.2.

Consider $V_5\cong\mathbb{C}^5$. Following [5], for $v\in V_5-\lbrace 0 \rbrace$, we call Pfaffian quadric P_v the symmetric bilinear form $x \mapsto vol (v\wedge x \wedge x )$ on $\bigwedge^2 V_5$.

The Grassmannian is the intersection of all the Pfaffian quadrics; since $\ker(P_v)=v\wedge V_5$, every Pfaffian quadric P_v has rank 6.

An automorphism α on a hyper-Kähler manifold X is symplectic if it acts trivially on a 2-form on X, non-symplectic otherwise. We always consider $H^2(X,\mathbb{Z})$ endowed with the Beauville–Bogomolov (BB) form. A $\langle 2t \rangle$-polarization is a polarization whose square is 2t with respect to the BB form.

We write $S^{[2]}$ for the Hilbert square on a $K3$ surface S. For any $H\in \operatorname{NS}(S)$, we denote by H ₂ the induced class on $S^{[2]}$ and by $\delta\in \operatorname{NS}(S^{[2]})$ the class of the divisor such that 2δ parametrizes non-reduced subschemes.

2. EPW sextics and GM varieties

2.1. EPW sextics

O’Grady provided slightly different points of view on the construction of EPW sextics, see for example [16, 19] and [18]; here we follow mainly [18], with a view to [23], by Debarre and Kuznetsov.

Let V ₆ be a six-dimensional vector space over $\mathbb{C}$, on which we fix a volume form $\bigwedge^6 V_6\xrightarrow{\sim} \mathbb{C}$, which in turns induces a symplectic form ω on $\bigwedge^3 V_6$.

In [7, example 9.3], Eisenbud, Popescu, and Walter introduced the Lagrangian subbundle F of $\mathcal{O}_{\mathbb{P}(V_6)}\otimes \bigwedge^3 V_6$, whose fiber over $[v]$ is $F_v=v\wedge \bigwedge^2 V_6$.

Given $v\in V_6-\lbrace 0 \rbrace$, we can fix a decomposition $V_6\cong \mathbb{C}v\oplus V_5$ for some hyperplane $V_5\subset V_6$, which induces $\bigwedge^3 V_6\cong \bigwedge^3 V_5 \oplus F_v$. Every element of F_v can be written in the form $v\wedge \eta$ for some $\eta\in \bigwedge^2 V_5$ and this induces an isomorphism of vector spaces

(1)\begin{align} \rho\colon F_v &\xrightarrow{\sim} \bigwedge^2 V_5 \end{align}

(2)\begin{align} v\wedge \eta &\mapsto \eta. \end{align}

This observation will be useful later.

We call $\mathbb{LG}(\bigwedge^3 V_6)\subset G(10,\bigwedge^3 V_6)$ the symplectic Grassmannian, parametrizing Lagrangian subspaces with respect to the symplectic form on $\bigwedge^3 V_6$; since two volume forms differ by a non-zero constant, $\mathbb{LG}(\bigwedge^3 V_6)$ does not depend on the choice of the volume form. From now on, $A\in \mathbb{LG}(\bigwedge^3 V_6)$ will be a Lagrangian subspace in $\bigwedge^3 V_6$ such that $\mathbb{P}(A)\cap G(3,V_6)=\emptyset$.

For a fixed A, we consider the map of vector bundles $\lambda_A\colon F\to \mathcal{O}_{\mathbb{P}(V_6)} \otimes A^\vee$ such that $(\lambda_A)_{[v]}(x)=\omega(x,-)_{|_{A}}$ for every $[v]\in \mathbb{P}(V_6)$. Since A is a Lagrangian subspace, the map is injective on $[v]\in \mathbb{P}(V_6)$ if and only if $F_v\cap A=0$. Since $\mathrm{rk}(F)=\mathrm{rk}(\mathcal{O}_{\mathbb{P}(V_6)}\otimes A^\vee)$, we can consider the map $ \operatorname{det} (\lambda_A)\colon \operatorname{det} (F)\to \operatorname{det} (\mathcal{O}_{\mathbb{P}(V_6)}\otimes A^\vee).$

Definition 2.1.

The determinantal variety $Y_A= Z( \operatorname{det} (\lambda_A))$ is a sextic hypersurface (see [16, (1.8)]), called EPW sextic.

We also associate with A a stratification of $\mathbb{P}(V_6)$: for $k\geq 0$ we define

(3)\begin{equation} Y^{\geq k}_A=\lbrace [v] \in \mathbb{P}(V_6) | \dim(F_v\cap A)\geq k \rbrace; \end{equation}

\begin{equation*}Y^{k}_A=\lbrace [v] \in \mathbb{P}(V_6) | \dim(F_v\cap A)= k \rbrace.\end{equation*}

The sets $Y^{\geq k}_A$ are degeneracy loci and are thus endowed with a natural structure of scheme. For every k, the variety $Y^{\geq k+1}_A$ is a closed subvariety of $Y^{\geq k}_A$ and clearly $Y_A=Y_A^{\geq 1}$.

Proposition 2.2.

If $\mathbb{P}(A)\cap G(3,V_6)=\emptyset$, then Y_A is integral and normal and $\operatorname{Sing}(Y_A)=Y^{\geq 2}_A$. Moreover, $Y^{\geq 2}_A$ is a normal integral surface whose singular locus is $Y^{3}_A$, which in turn is finite; when smooth, the surface $Y^{\geq 2}_A$ is of general type. Finally $Y^{\geq 4}_A=\emptyset$.

Proof. Integrality and normality for Y_A follow from $\operatorname{Sing}(Y_A)=Y^{\geq 2}_A$. For the rest, see [18, corollary 2.5], [18, proposition 2.9], [19, claim 3.7], [22, theorem B.2], and [3, proposition 1.10].

The situation can be much more complicated for EPW sextics associated with a Lagrangian subspace A which contains a non-zero decomposable vector; however, for our scope, we will not need to deal with them.

2.2. The dual EPW sextic

One interesting feature of EPW sextics is that they admit a dual counterpart. The volume form on V ₆ induces a volume form on $ V^\vee_6$ too, and thus a symplectic form on $\bigwedge^3 V^\vee_6\cong (\bigwedge^3 V_6)^\vee$, hence for $A\in \mathbb{LG}(\bigwedge^3 V_6)$ we also have

\begin{equation*}A^\perp=\lbrace \phi\in (\bigwedge^3 V_6)^\vee| \phi(a)=0 \,\mbox{for every }a\in A\rbrace\in \mathbb{LG}((\bigwedge^3 V_6)^\vee);\end{equation*}

$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$ if and only if $\mathbb{P}(A^\perp)\cap G(3,V^\vee_6)=\emptyset$, see [18, Section 2.6].

Definition 2.3.

We call dual EPW sextic the EPW sextic associated with $A^\perp$.

When $\mathbb{P}(A)\cap G(3,V_6)=\emptyset$, the hypersurfaces $Y_A\subset \mathbb{P}(V_6)$ and $Y_{A^\perp}\subset \mathbb{P}(V^\vee_6)$ are projectively dual; this has been proved by O’Grady, see also [22, proposition B.3].

As showed in [18, Section 2.6], the EPW strata associated with $A^\perp$ can also be described as

\begin{equation*}Y^{\geq k}_{A^\perp}=\lbrace [V_5] \in \mathbb{P}(V^\vee_6) | \dim(\bigwedge^3 V_5 \cap A)\geq k \rbrace\end{equation*}

Definition 2.4.

We denote by

\begin{equation*}\Sigma=\lbrace A\in \mathbb{LG}(\bigwedge^3 V_6) | \mathbb{P}(A)\cap G(3,V_6)\neq \emptyset \rbrace \end{equation*}

the set of Lagrangian subspaces A admitting a non-zero decomposable vector, by

\begin{equation*}\Delta=\lbrace A\in \mathbb{LG}(\bigwedge^3 V_6) | Y^{\geq 3}_A\neq \emptyset\rbrace\end{equation*}

the set of Lagrangian subspaces whose third stratum is not empty, and by

\begin{equation*}\Pi=\lbrace A\in \mathbb{LG}(\bigwedge^3 V_6) | Y^{\geq 3}_{A^\perp}\neq \emptyset\rbrace \end{equation*}

the set of Lagrangian subspaces $A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}$ such that the third stratum associated with the dual Lagrangian $A^\perp$ is not empty.

Following O’Grady, we write $\mathbb{LG}(\bigwedge^3 V_6)^0=\mathbb{LG}(\bigwedge^3 V_6)- (\Delta\cup \Sigma)$.

The subsets Σ, $\Delta, \Pi$ descend to distinct irreducible divisors in $\mathbb{LG}(\bigwedge^3 V_6)//$ $PGL(6)$, see [18, proposition 3.1, Item 1] and [19, proposition 2.2]. We denote their quotients in the same way.

Proposition 2.5. [22], proposition B.8

If $A\notin \Sigma$, the automorphism group of Y_A is $\lbrace \alpha\in \operatorname{PGL}(V_6)$ such that $ \left(\bigwedge^3 \alpha\right)(A)=A \rbrace$. In particular, every $\alpha\in \operatorname{Aut}(Y_A)$ fixes $Y_A^k$ for every $k\geq 0$.

From now on we always identify $PGL(V_6)$ and $PGL(V^\vee_6)$ through the natural isomorphism sending $\phi\in \operatorname{PGL}(V_6)$ to $(\phi^{-1})^\vee$, which maps $[V_5]$ to $[\phi(V_5)]$.

Corollary 2.6.

If $A\notin \Sigma$, then $\operatorname{Aut}(Y_A)=\operatorname{Aut}(Y_{A^\perp})$.

2.3. Double EPW sextics

The importance of EPW sextics stems mostly from the fact that, in the general case, they admit a ramified double cover which is a hyper-Kähler manifold, as was proved by O’Grady in [16]. We also refer to [23, theorem 5.2, Item 1].

We denote by $\mathcal{R}=\operatorname{Coker} (\lambda_A)_{|_{Y_A}}$ the first Lagrangian cointersection sheaf; it is a reflexive sheaf, see [16, proposition 4.3, Item 1].

Theorem 2.7. (O’Grady)

Consider $A\in \mathbb{LG}(\bigwedge^3 V_6)$ such that $\mathbb{P}(A)\cap G(3,V_6)=\emptyset$. Then there is a unique double cover $f_A\colon X_A\to Y_A$ with branch locus $Y^{\geq 2}_A$ such that

(4)\begin{equation} (f_A)_*\mathcal{O}_{X_A}\cong \mathcal{O}_{Y_A} \oplus \mathcal{R}(-3). \end{equation}

The variety X_A is integral and normal, and its singular locus is $f^{-1}_A(Y^3_A)$.

A double cover of Y_A can be constructed even when $\mathbb{P}(A)\cap G(3,V_6)\neq \emptyset$, provided that $Y_A\neq \mathbb{P}(V_6)$. However, in this case, X_A is never smooth.

Definition 2.8.

We call double EPW sextic the double cover X_A, and we denote by ι_A the associated covering involution.

Theorem 2.9 [19], theorem 4.25

Suppose that X_A is smooth, i.e. $A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0$. Then X_A is a hyper-Kähler manifold equivalent by deformation to the Hilbert square on a K3 surface. The involution ι_A is non-symplectic.

The ample class $D_A=f^*_A\mathcal{O}_{Y_A}(1)\in \operatorname{NS}(X_A)$ is the only primitive polarization on X_A coming from Y_A, by Lefschetz hyperplane theorem. It has square 2 with respect to the BB form; the general element, inside the moduli space $\mathcal{M}_2$ (see appendix A for a definition) is isomorphic to $(X_A,D_A)$ for some $A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0$.

We denote by $\operatorname{Aut}_{D_A}(X_A)\subseteq \operatorname{Aut}(X_A)$ the group of automorphisms fixing D_A, or equivalently commuting with ι_A. There is a short exact sequence

(5)\begin{equation} 1\to \lbrace \operatorname{id},\iota_A \rbrace\to \operatorname{Aut}_{D_A}(X_A) \to \operatorname{Aut}(Y_A) \to 1. \end{equation}

Proposition 2.10. [29], proposition A.2

The sequence (5) splits, so $\operatorname{Aut}_{D_A}(X_A)\cong \operatorname{Aut}(Y_A)\times \lbrace \operatorname{id},\iota_A \rbrace$. When X_A is smooth, under this isomorphism $\operatorname{Aut}(Y_A)$ is the group of symplectic automorphisms fixing the polarization D_A.

2.4. GM varieties

In this section, we introduce GM varieties and in the next one we explain how they admit an associated EPW sextic.

We fix a five-dimensional complex vector space V ₅. For the following definition and results, we refer to [22, definition 2.1].

Definition 2.11.

Consider a vector subspace $W\subseteq \bigwedge^2 V_5$ of dimension $\dim W\geq 6$ and a quadric hypersurface $Q\subset \mathbb{P}(W)$. We call ordinary GM intersection the scheme

\begin{equation*} Z= \mathbb{P}(W)\cap G(2,V_5)\cap Q . \end{equation*}

It is an ordinary GM variety if Z is integral and $\dim(Z)=\dim(W)-5$.

When Z is a GM variety, it is a complete intersection inside $G(2,V_5)$ and has degree 10 in $\mathbb{P}(W)$.

If $\dim(Z)\geq 3$, Z is a Fano variety of index n − 2 (see for example [22, theorem 2.3]). If $\dim(Z)=2$, let H be the polarization given by $\mathcal{O}_{\mathbb{P}(W)}(1)$: the polarized variety (Z, H) is a Brill–Noether general $K3$ surface, see [30], also [4, theorem 10.3]. The converse holds under a technical condition, which is strongly smoothness.

Definition 2.12.

Let $Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be an ordinary n-dimensional GM variety. The Grassmannian hull of Z is the $(n+1)$-dimensional intersection $M_Z=\mathbb{P}(W)\cap G(2,V_5)$. We say that Z is strongly smooth if both Z and M_Z are dimensionally transverse and smooth.

Although GM curves exist, here we never deal with them, since they are never strongly smooth or, equivalently, their associated double EPW sextics are never smooth, as we will see later (theorem 2.17).

If Z is strongly smooth, its Grassmannian hull has Picard rank 1 by the Lefschetz hyperplane theorem. For every Brill–Noether general $K3$ surface (S, H), the projective model $\phi_{|H|}(S)$ is a smooth ordinary GM variety of dimension 2, provided that it is strongly smooth (see remark 2.19), whereas smoothness and strongly smoothness are equivalent when $n\geq 3$.

Definition 2.13.

An isomorphism between $Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ and $Z^{\prime}=\mathbb{P}(W^{\prime})\cap G(2,V^{\prime}_5)\cap Q^{\prime}$ is a linear map $\phi \colon \mathbb{P}(W)\to \mathbb{P}(W^{\prime})$ such that $\phi(Z)=Z^{\prime}$. We denote by $\operatorname{Aut}(Z,\mathbb{P}(W))$ the group of automorphisms of Z.

By the Lefschetz hyperplane theorem, a smooth GM variety Z of dimension at least 3 has Picard rank 1. So, in this case, $\operatorname{Aut}(Z,\mathbb{P}(W))$ is the whole automorphism group of Z as an abstract variety.

Debarre and Kuznetsov provided an intrinsic characterization of normal GM varieties in [22, theorem 2.3]; this leads to the definition of GM data, which are a set of linear data which can be associated with any normal GM variety. These collections of objects are very useful to handle.

Definition 2.14.

Ordinary GM data $(W,V_6,V_5,\mu,q,\epsilon)$ of dimension n consists of (we set $L=(V_6/V_5)^\vee$ for readability)

• • a $(n+5)$-dimensional vector space W;

• • a six-dimensional vector space V ₆;

• • a hyperplane V ₅ of V ₆;

• • an injective linear map $\mu\colon W\otimes L \hookrightarrow \bigwedge^2 V_5$;

• • a linear map $q\colon V_6\to S^2 W^\vee$;

• • a linear isomorphism $\epsilon\colon \bigwedge^5 V_5\to L^{\otimes 2}$,

such that, for all $v\in V_5$, $w_1, w_2\in W$,

(6)\begin{equation} q(v)(w_1,w_2)=\epsilon(v\wedge \mu(w_1)\wedge \mu(w_2)). \end{equation}

Definition 2.15.

An isomoprhism between two ordinary GM data sets $(W,V_6,V_5,\mu,q,\epsilon)$ and $(W^{\prime},V^{\prime}_6,V^{\prime}_5,\mu^{\prime},q^{\prime},\epsilon^{\prime})$ is a triple of linear isomorphisms $\phi_W\colon W\to W^{\prime}$, $\phi_V\colon V_6\to V^{\prime}_6$, $\phi_L\colon L\to L^{\prime}$ such that $\phi_V(V_5)=V^{\prime}_5$, $\epsilon^{\prime}\circ \bigwedge^5 \phi_V=\phi^{\otimes 2}_L\circ \epsilon$, and the following diagrams commute

Debarre and Kuznetsov showed in [22, Section 2.1] how to associate a set of GM data $(W,V_6,V_5,\mu,q,\epsilon)$ to a normal GM variety and to GM data a GM intersection which a priori may not be a normal variety; when it is the case, the two constructions are mutually inverse and behave well with respect to the definition of isomorphisms. To keep the exposition readable, we refer to their article for the explicit correspondence.

Finally, we point out that another class of GM intersections, special GM intersections, exist, see for example [22, Section 2.5].

2.5. A correspondence between data sets

Debarre and Kuznetsov defined another set of data. Every time we consider a six-dimensional complex vector space V ₆, we endow it with a volume form, which induces a symplectic form on $\bigwedge^3 V_6$.

Definition 2.16.

An ordinary Lagrangian datum is a collection $(V_6,V_5,A)$, where

• • V ₆ is a six-dimensional complex vector space,

• • $V_5\subset V_6$ is a hyperplane,

• • $A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}$ is a Lagrangian subspace.

The ordinary Lagrangian data $(V_6,V_5,A)$ and $(V^{\prime}_6,V^{\prime}_5,A^{\prime})$ are isomorphic if there is a linear isomorphism $\phi:V_6\to V^{\prime}_6$ such that $\phi(V_5)=V^{\prime}_5$ and $(\bigwedge^3 \phi)(A)=A^{\prime}$.

The next result, by Debarre and Kuznetsov, sums up the results by O’Grady, [16, Section 5 and Section 6], [19, Section 4], in dimension 2 and Iliev and Manivel, [5, Section 2] in dimension 5 and [5, Section 4] in dimension $3,4$.

Theorem 2.17.

For $n\in \lbrace 1,\ldots,5 \rbrace$, there is a bijection between the set of iso-morphism classes of Lagrangian data sets $(V_6,V_5,A)$, with

• • $\mathbb{P}(A)\cap G(3,V_6)=\emptyset$

• • $[V_5]\in Y^{5-n}_{A^\perp}$

and isomorphism classes of strongly smooth ordinary GM varieties of dimension n. In particular, there are no strongly smooth ordinary GM curves.

Proof. See [22, theorem 3.6] and [22, theorem 3.16]. For n = 1, recall that $A^\perp$ admits non-zero decomposable elements if and only if A does. Then we only need to observe that $Y^{\geq 4}_{A^\perp}=\emptyset$ by proposition 2.2.

It is interesting to note that the construction works for non-strongly smooth GM varieties, but in that case A always admits non-zero decomposable elements.

Definition 2.18.

Given some Lagrangian data $(V_6,V_5,A)$, the associated ordinary GM variety is the GM intersection Z obtained by theorem 2.17 from $(V_6,V_5,A)$. When not otherwise specified, we write $Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ where Q is the projective quadric associated with q(x) for any $x\in V_6-V_5$.

Given a strongly smooth ordinary GM variety Z, we denote by A(Z) the Lagrangian subspace associated with Z by theorem 2.17.

Remark 2.19.

Strongly smooth special GM varieties of dimension 2 do not exist by [22, remark 3.17] and proposition 2.2.

For any $(V_6,V_5,A)$ and $p\geq 2$, there is a short exact sequence

(8)\begin{equation} 0 \to \bigwedge^p V_5 \to \bigwedge^p V_6 \xrightarrow{\lambda_p} \bigwedge^{p-1} V_5\otimes L^\vee \to 0. \end{equation}

The following two remarks will be useful in the proof of theorem 3.2 in the frame of the study of two-dimensional GM varieties: although the relation between the latter and EPW sextics is already clear in [16] and [19], the use of GM data allows a very precise study.

Remark 2.20.

Given some Lagrangian data $(V_6,V_5,A)$, theorem 2.17 allows to describe explicitly the associated ordinary GM variety $Z\subseteq \mathbb{P}(\bigwedge^2 V_5)$. We fix $x\in V_6-V_5$, hence an isomorphism $\bigwedge^3 V_6 \cong \bigwedge^3 V_5\oplus F_x$. Under this isomorphism and (1), the sequence (8) for p = 3 becomes

(9)\begin{equation} 0 \to \bigwedge^3 V_5 \to \bigwedge^3 V_6\cong \bigwedge^3 V_5\oplus F_x \xrightarrow{\lambda_3} \bigwedge^{2} V_5 \to 0 . \end{equation}

For p = 4, the sequence (8) can be rewritten as $ 0 \to \bigwedge^4 V_5 \to \bigwedge^4 V_6 \xrightarrow{\lambda_4} \bigwedge^{3} V_5 \to 0 $ as well. We will thus consider W as a subspace of $\bigwedge^2 V_5$ and we identify $\bigwedge^5 V_5$ with $\mathbb{C}$ through the volume form on V ₆. Then $W=\lambda_3(A)$. As for the quadric hypersurface, we call Q(x) the projective quadric hypersurface associated with $q(x)\in \operatorname{Sym}^2W^\vee$. We have $Z=M_Z\cap Q(x)$. Consider $ w\in W$: since $W=\lambda_3(A)$, there is $\eta\in \bigwedge^3 V_5$ such that $\eta+x\wedge w\in A$. By definition of $q: V_6\to \operatorname{Sym}^2 W^\vee$ given in theorem 2.17, the element $w=\lambda_3(\eta+x\wedge w)$ lies in Q(x) if and only if $\eta\wedge w=\lambda_4(x\wedge \eta)\wedge w=0$.

Remark 2.21.

Consider instead some ordinary GM intersection $Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$, with associated GM data $(W,V_6,V_5,\mu,q,\epsilon)$. We consider again $W\subseteq \bigwedge^2 V_5\otimes L^\vee$ and $x\in V_6- V_5$ such that $Q=Q(x)$. As above $\bigwedge^3 V_6 \cong \bigwedge^3 V_5\oplus F_x$ and (9) holds. Also $\bigwedge^3 V_5\cong (\bigwedge^2 V_5)^\vee$ via $\epsilon:\bigwedge^5 V_5\to L^{\otimes 2}$ and A(Z) is

\begin{align*} \Bigg\{ (\eta,w)\in \bigwedge^3 V_5\oplus W | w\in W , \epsilon(\eta\wedge \mu(\_ ))_{|_W}& = -(q(x)(w,-)\otimes [x])\\ &\quad\mbox{ as elements of } W^\vee \Bigg\} .\end{align*}

Now we forget ϵ, we consider again $W\subseteq \bigwedge^2 V_5$ and $\bigwedge^3 V_5$ as $(\bigwedge^2 V_5)^\vee$. We have then a splitting sequence $0\to W^\perp \to (\bigwedge^2 V_5)^\vee \to W^\vee \to 0$. Putting all together, we can rewrite $A(Z)\subset W^\perp \oplus W^\vee\oplus F_x$ as

(10)\begin{equation} \left\lbrace (\xi,-q(x)(w,-),x\wedge w)\in W^\perp \oplus W^\vee\oplus F_x \quad |\quad \xi\in W^\perp, w\in W \right\rbrace . \end{equation}

From this description, under the identification $\bigwedge^3 V_5\cong (\bigwedge^2 V_5)^\vee$, we have $A(Z)\cap \bigwedge^3 V_5=W^\perp$ by [22, proposition 3.13, Item a].

Consider the linear map $x\wedge W\to W^\vee$ that sends $x\wedge w$ to $-q(x)(w,-)$: its graph $\Gamma_x\subset W^\vee\oplus W$ induces a decomposition $A(Z)=W^\perp\oplus \Gamma_x\subset W^\perp \oplus W^\vee\oplus F_x$.

3. $K3$ surfaces whose associated double EPW sextic is smooth

We denote by $\mathcal{K}_{10}$ the moduli space of $\langle 10 \rangle$-polarized $K3$ surfaces.

Definition 3.1.

The divisor $D_{x,y}\subset \mathcal{K}_{10}$ is the locus of pairs (S, H) such that there exists a primitive sublattice $\mathbb{Z}H+\mathbb{Z}D\subseteq \operatorname{NS}(S)$ whose Gram matrix is $$\left[\begin{array}{cc} 10 & x \\ x & y \\ \end{array}\right].$$

In this section, we want to find precise conditions on a Brill–Noether general, $\langle 10 \rangle$-polarized, $K3$ surface such that the associated double EPW sextic $X_{A(S)}$ is a hyper-Kähler manifold.

We answer this question in two ways, as a condition on curves on the embedded $K3$ surface (theorem 3.2) and as a divisorial condition on $\mathcal{K}_{10}$ (theorem 3.3). In particular, thanks to theorem 3.3, finding whether $X_A(S)$ is smooth becomes a lattice-theoretic problem on $\operatorname{NS}(S)$.

Theorem 3.2.

Let $S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be a $\langle 10\rangle$-polarized $K3$ surface. The double cover $X_{A(S)}$ of the associated EPW sextic $Y_{A(S)}$ is smooth if and only if S is strongly smooth and contains neither lines nor quintic elliptic pencils.

We recall that $(S,H)\in \mathcal{K}_{10}$ is Brill–Noether general if and only if the projective model of (S, H) is a smooth ordinary GM variety of dimension 2.

From now on, we see a Brill–Noether general $(S,H)\in \mathcal{K}_{10}$ as a smooth ordinary GM two-dimensional variety $S = \mathbb{P}(W) \cap G(2,V_5) \cap Q$, with Grassmannian hull $M_S=\mathbb{P}(W) \cap G(2,V_5)$.

Theorem 3.3.

Let $(S,H)\in \mathcal{K}_{10}$ be a $\langle 10 \rangle$-polarized $K3$ surface. If (S, H) is Brill–Noether general, let $A(S)\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}$ be the Lagrangian subspace associated with (S, H). Then

1. (1) (S, H) is Brill–Noether general if and only if $(S,H)\notin D_{h,0}$ for $h\in \lbrace 1,2,3 \rbrace$;

2. (2) if (S, H) is Brill–Noether general, then (S, H) is strongly smooth if and only if $A(S)\notin \Sigma$, if and only if $(S,H)\notin D_{4,0}$.

Moreover, if (S, H) is strongly smooth, then $A(S)\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0$ if and only if $(S,H)\notin D_{x,y}$, with $(x,y)\in \lbrace(1,-2),(5,0)\rbrace$. In particular

1. (3) $Y^3_{A(S)}\cap \mathbb{P}(V_5)=\emptyset$ if and only if $(S,H)\notin D_{1,-2}$;

2. (4) $Y^3_{A(S)} - \mathbb{P}(V_5)=\emptyset$ if and only if $(S,H)\notin D_{5,0}$.

Grassmannian hulls M_S of strongly smooth Brill–Noether general $K3$ surfaces are isomorphic, see [13] or [19, proposition 5.2, Item 3)]. We need a characterization of the lines inside M_S. For $v\in V_5-\lbrace 0 \rbrace$ and $v\in V_3\subset V_5$ with $V_3\cong \mathbb{C}^3$, we set $L_{v,V_3}=\lbrace v\wedge t |t\in V_3 \rbrace$. Every line in $G(2,V_5)$, hence in M_S, is of the form $\mathbb{P}(L_{v,V_3})$.

Lemma 3.4.

Consider $v\in V_5-\lbrace 0 \rbrace$. There exists a three-dimensional vector space $V_3\ni v$ such that the line $\mathbb{P}(L_{v,V_3})\subset G(2,V_5)$ is contained in M_S if and only if $\ker(P_v)\cap W$ has dimension exactly 2. In this case, $L_{v,V_3}=\ker(P_v)\cap W$.

Proof. We fix a decomposition $V_5= \mathbb{C}v\oplus U$; we know that $\ker(P_v)=v\wedge U$, hence $L_{v,V_3}\subset\ker(P_v)$ for any $V_3\ni v$. If $\ker(P_v)\cap W$ has dimension 2, its projectivization is a line on M_S of the form $\mathbb{P}(L_{v,V_3})$. Conversely, if $\mathbb{P}(L_{v,V_3})\subset M_S$ then $L_{v,V_3}\subset W\cap \ker(P_v)$, in particular $\dim(W\cap \ker(P_v))\geq 2$. The dimension is exactly 2, or M_S would contain a plane, which is absurd since M_S has Picard rank 1.

When a GM variety Z is strongly smooth, for $v\in V_5-\lbrace 0\rbrace$, the kernel of the Pfaffian quadric $(P_v)_{|_W}$ is $\ker(P_v)\cap W$ and its corank is at least $\dim(W)-6$. For $x\notin V_6$, the kernel of q(x) is computed in [22, proposition 3.13, Item b].

Proof of theorem 3.2

The proof consists of three parts. We already know, from §2.5, that strongly smoothness is a necessary condition to have $X_{A(S)}$ smooth. We need to prove $A(S)\notin \Delta$, i.e. $Y^3_{A(S)}=\emptyset$.

S contains no line if and only if $Y^3_{A(S)}\cap \mathbb{P}(V_5)=\emptyset$. Fix $x\in V_6-V_5$, so that $S=M_S\cap Q(x)$, with Q(x) the projective quadric associated with q(x); as in remark 2.20, we have $\bigwedge^3 V_6\cong (\bigwedge^2 V_5)^\vee\oplus F_x$ and a linear map $\lambda_3 \colon (\bigwedge^2 V_5)^\vee\oplus F_x \to \bigwedge^2 V_5$ sending $(\phi, x\wedge w)$ to w.

For $v\in V_5-\lbrace 0 \rbrace$, we fix a decomposition $V_5=\mathbb{C} v \oplus U$. Since $F_v=(v\wedge \bigwedge^2 U) \oplus v\wedge U$, we can write

\begin{equation*} \lambda_3(A)=W \quad\quad \lambda_3(F_v)=v\wedge U \quad\quad \ker((P_v)_{|_W})=(v\wedge U)\cap W . \end{equation*}

We have $A\cap (\bigwedge^2 V_5)^\vee=W^\perp$ by remark 2.21; we can also write $F_v\cap (\bigwedge^2 V_5)^\vee =v\wedge \bigwedge^2 U=\ker(P_v)^\perp$ and by (9),

(11)\begin{equation} \dim(A\cap F_v)=\dim (\lambda_3 (A\cap F_v))+\dim(A\cap F_v\cap (\bigwedge^2 V_5)^\vee). \end{equation}

We have $\lambda_3 (A\cap F_v)\subseteq \ker(P_v) \cap W$. The dimension of $\ker(P_v) \cap W$ is at most 2, see lemma 3.4, in particular:

1. (1) If $\dim(\ker(P_v) \cap W)=2$, then $\ker(P_v)+W$ has dimension 9, hence there is only one hyperplane containing both of them, so $\dim(W^\perp\cap \ker(P_v)^\perp)=1$;

2. (2) If $\dim(\ker(P_v) \cap W)=1$, then $\ker(P_v)+W=\bigwedge^2 V_5$ and $W^\perp\cap \ker(P_v)^\perp=0$.

By (11), since $A\cap F_v\cap (\bigwedge^2 V_5)^\vee= W^\perp\cap \ker(P_v)^\perp$ in order to obtain $\dim(A\cap F_v)=3$, we must consider all the $v\in V_5$ for which (1) above holds. For such a v, we have $\lambda_3(A\cap F_v)=\lambda_3(A)\cap \lambda_3(F_v)$, so that the RHS of (11) can be 3. The line $\ell=\mathbb{P}(\lambda_3(A\cap F_v))$ lies in M_S, so we only need to prove $\ell\subset Q(x)$. By remark 2.20, we know that $q(x)(w,w)=0$ if and only if there exists $\eta\in \bigwedge^3 V_5$ such that $\eta+x\wedge w\in A$ and $\eta\wedge w=0$. Consider any $v\wedge u$ inside $\lambda_3(A\cap F_v)$: by hypothesis, there exists $\eta\in \bigwedge^3 V_5$ such that $\eta+x\wedge v\wedge u\in A\cap F_v$. This means that $\eta\in F_v$, so $\eta\wedge v\wedge u=0$. In particular $v\wedge u\in Q(x)$, hence the whole line $\mathbb{P}(\lambda_3(A\cap F_v))$ is contained in S.

On the other hand, if $\mathbb{P}(L_{v,V_3})\subset S$ for some $v\in V_5$, by lemma 3.4, we also have $L_{v,V_3}=\ker(P_v)\cap W\cong \mathbb{C}^2$, hence there is some non-zero $\eta\in W^\perp\cap \ker(P_v)^\perp$. Then to prove $[v]\in Y^3_A\cap \mathbb{P}(V_5)$, we need $\lambda_3(A\cap F_v)=\lambda_3(A)\cap \lambda_3(F_v)=W\cap\ker(P_v)$.

Consider $v\wedge u\in W\cap\ker(P_v)$. We use (10): under the decomposition $(\bigwedge^2 V_5)^\vee \cong W^\perp\oplus W^\vee$, the element

\begin{equation*}\alpha=(0,-q(x)(v\wedge u,-),x\wedge v\wedge u)\in W^\perp\oplus W^\vee \oplus F_x\end{equation*}

lies in A and $\lambda_3(\alpha)=v\wedge u$. To prove that $\alpha\in F_v$, so that we have $v\wedge u\in \lambda_3(A\cap F_v)$, we show $\alpha\wedge \eta=0$ for every $\eta\in F_v$, then we can conclude, since F_v is Lagrangian. As α is zero on the first component, we are left to prove $0=\alpha\wedge \eta=-q(x)(v\wedge u,\eta)$ for $\eta\in F_v\cap W=L_{v,V_3}$. By hypothesis $\mathbb{P}(L_{v,V_3})\subset S\subset Q(x)$: this happens if and only if $q(x)_{|_{L_{v,V_3}\times L_{v,V_3}}}=0$ and in particular $q(x)(v\wedge u,-)=0$ on $L_{v,V_3}$.

If $Y^3_{A(S)}-\mathbb{P}(V_5)\neq \emptyset$, then S contains a quintic elliptic pencil. Consider $[x]\in Y^3_A - \mathbb{P}(V_5)$, so that S is the transverse intersection of M_S and Q(x): by [22, proposition 3.13, Item b], the corank of q(x) is 3.

The quadric Q(x) is a cone of $\mathbb{P}(\ker (q(x)))=\mathbb{P}(A\cap F_x)$ over a smooth quadric surface $\mathcal{Q}\subset\mathbb{P}^3$; the latter has two families of lines on it and two lines inside $\mathcal{Q}$ intersect if and only if they do not lie in the same family. Let $\lbrace\ell_t\rbrace_{t\in \mathbb{P}^1}$ be one of the two families. The cone of $\mathbb{P}(\ker (q(x)))$ over $\ell_t$ is a $\mathbb{P}^4$ which we denote by π_t. By transversality, the intersection $M_S \cap \pi_t = S\cap \pi_t$ is a degree-5 curve on S.

Consider now $(\pi_t\cap S)$ and $(\pi_{t^{\prime}}\cap S)$ for $t\neq t^{\prime}$. We have $\pi_t\cap \pi_{t^{\prime}}=\mathbb{P}(\ker (q(x)))$, as $\ell_t$ and $\ell_{t^{\prime}}$ are in the same family, so $(\pi_t\cap S)\cap (\pi_{t^{\prime}}\cap S)=\mathbb{P}(\ker (q(x)))\cap S$. But the latter is empty, since $\mathbb{P}(\ker (q(x)))=\operatorname{Sing}(Q(x))$ and S is smooth. So the pencil $\lbrace S\cap \pi_t\rbrace_{t\in \mathbb{P}^1}$ is elliptic.

If S contains a quintic elliptic pencil, then $Y^3_{A(S)}-\mathbb{P}(V_5)\neq \emptyset$. We call $E\in \operatorname{NS}(S)$ the class of the elliptic pencil: the sublattice $$\mathbb{Z} H+\mathbb{Z} E=K\subset \operatorname{NS}(S)$$ has Gram matrix $$\begin{pmatrix} 10 & 5\\ 5 & 0 \end{pmatrix}$$. The orthogonal complement of H in K is generated by $\kappa=H-2E$; we denote by κ ₂ the corresponding class in $\operatorname{NS}(S^{[2]})$.

From now on, we consider notations and results from appendix A. Following that, we call $e,f$ two canonical generators of a copy of U; moreover, $e_1,f_1$ will be the canonical generators of a second copy of U, orthogonal to the first one. By Eichler Criterion [33, proposition 3.3], there is an isometry $\psi:H^2(S^{[2]},\mathbb{Z})\to \Lambda$ such that $\psi(H_2-2\delta)=h=e+f$ and $\psi(\kappa_2)=g+2\ell-2e_1-8f_1$; the involution j on the moduli space exchanges g and $\ell$ in $\Lambda_h$, so that $j(\kappa_2)=g+2\ell-2e_1-8f_1$ is an algebraic class for $X_{A(S)}$, which lies in $D_A^\perp$. It has square 10 and divisibility 2 in $H^2(X_{A(S)},\mathbb{Z})$, so D_A lies in a flopping wall and is not ample, see [28, theorem 5.1, Item b].

Hence $Y_{A(S)}^3\neq \emptyset$ and more precisely $Y_{A(S)}^3 - \mathbb{P}(V_5)\neq \emptyset$, since points in $Y_{A(S)}^3 \cap \mathbb{P}(V_5)$ correspond to lines in S, hence sublattices in $\operatorname{NS}(S)$ whose Gram matrix is $$\begin{pmatrix} 10 & 1\\ 1 & -2 \end{pmatrix}$$. It is then sufficent to observe that such a sublattice is neither an overlattice nor a sublattice of $H\mathbb{Z}+E\mathbb{Z}$.

We can now prove the second version of the characterization.

Proof of theorem 3.3

For (1) see [9, lemma 2.8]. We do not need to ask $(S,H)\notin D_{5,2}$, as in [9], since this is a divisor inside the complement of $$\mathcal{K}_{10}$$ in its closure; to see that, one checks that the orthogonal complement of H in $\mathbb{Z} H+\mathbb{Z} D$ has square −2 and concludes by [35, theorem 2.7]. For (2) see [9, lemma 2.7]. The rest of the statement comes directly from theorem 3.2. Lines on $S=\mathbb{P}(W)\cap G2,V_5)\cap Q$ correspond to sublattices of $\operatorname{NS}(S)$ whose Gram matrix is $$\left[\begin{array}{cc} 10 & 1 \\ 1 & -2 \end{array}\right]$$, To prove (3), it is then sufficient to show that this lattice admits no non-trivial overlattice. For (4), the proof is the same, with the lattice whose Gram matrix is $$\left[\begin{array}{cc} 10 & 5 \\ 5 & 0 \end{array}\right]$$.

Finally, the description of lines in S can be made more precise.

Proposition 3.5.

Let $S= \mathbb{P}(W) \cap G(2,V_5) \cap Q$ be a strongly smooth $K3$ surface. Let A(S) be the associated Lagrangian subspace. Then

\begin{equation*} |Y^3_{A(S)}\cap \mathbb{P}(V_5)|= |\lbrace \ell \subset S| \ell\ \mbox{line}\rbrace|. \end{equation*}

Proof. This is an easy consequence of the first part of the proof above. To show the one-to-one correspondence, one observes that two lines $\mathbb{P}(L_{v,V_3})$ and $\mathbb{P}(L_{v^{\prime},V^{\prime}_3})$ intersect in at most one point if $[v]\neq [v^{\prime}]$.

4. Inducing automorphisms on double EPW sextics

We explain here how we can use our main result, and particularly theorem 3.3, to deduce the existence of automorphisms on families of smooth double EPW sextics. This was actually the first motivation for our study of strongly smooth GM surfaces, since the automorphisms of S act on the associated EPW sextic.

Proposition 4.1. Debarre and Kuznetsov

For Z a strongly smooth GM variety, there is a natural inclusion of $\operatorname{Aut}(Z,\mathbb{P}(W))$ in $\operatorname{Aut}(Y_{A(Z)})\cong \operatorname{Aut}(Y_{A(Z)^\perp})$ as the stabilizer of $[V_5]\in Y_{A(Z)^\perp}$.

Proof. See [22, proposition 3.21, Item c]: the result still holds if we replace ‘dimension $\geq 3$’ in the statement with ‘X strongly smooth’.

O’Grady showed that, for $S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ very general, the associated double EPW is smooth; however, the group $\operatorname{Aut}(S,\mathbb{P}(W))$ is trivial for S very general! In addition, all those automorphisms turn out to be symplectic, see propositions 2.10 and 5.2, and families of $K3$ surfaces in $\mathcal{K}_{10}$ carrying symplectic actions typically have big codimension.

So, to find automorphisms on some S that induce automorphisms on a smooth double EPW sextic, the tricky part is to control the generality. By theorem 3.3, we can easily deduce the existence of automorphisms on some smooth double EPW sextic, appealing to the classification of automorphisms on $K3$ surfaces. Let $(V_6,V_5,A)$ be the Lagrangian data associated with S: by theorem 2.17, $[V_5]\in Y^3_{A^\perp}$, hence A always lies in Π.

In the following statement, we denote by D_n the dihedral group of order 2n. In the proof, we use some results from the next section; we decided to postpone them to keep the exposition more cohesive.

Proposition 4.2.

Let G be one of the following groups,

\begin{equation*} \mathbb{Z}/n\mathbb{Z} \,\mbox{for }n\in\lbrace 2,3,4 \rbrace,\quad (\mathbb{Z}/2\mathbb{Z})^2, \quad D_n\,\mbox{for }n\in\lbrace 4,5,6\rbrace. \end{equation*}

There is a family of Lagrangian subspaces in $\Pi - (\Sigma \cup \Delta)$ such that, for any A in the family, the associated double EPW sextic X_A is smooth and admits a symplectic action of G which commutes with the covering involution.

For a general element of the family, $\operatorname{Aut}_{D_{A}}(X_{A})=\iota_{A}\times G$.

Proof. A $\langle 10 \rangle$-polarized $K3$ surface (S, H) admits a symplectic action of G which fixes H if and only if its Néron-Severi group contains $\mathcal{L}_G=\mathbb{Z} H\oplus \Omega_G$, where $\Omega_G$ is a negative-definite lattice associated with G, which turns out to be the coinvariant lattice for the action of G in cohomology. This has been proved by Nikulin [31], but see also [1, proposition 6.3].

We start by considering $G=\mathbb{Z}/n\mathbb{Z}$ with $n=2,\ldots 6$, $(\mathbb{Z}/2\mathbb{Z})^2$ or D ₄ and we want to prove that there is a family of $K3$ surfaces for which the very general element S has $\operatorname{NS}(S)=\mathcal{L}_G$. Then we will have $G=\operatorname{Aut}(S,\mathbb{P}(W))$ for $G\neq \mathbb{Z}/n\mathbb{Z}$ with $n=5,6$, and $\operatorname{Aut}(S,\mathbb{P}(W))=D_n$ otherwise: in the latter case, we appeal to [2, propositions 8.1 and 9.1] and then proposition 5.2, in the former simply to proposition 5.2.

By the surjectivity of the period map for marked $K3$ surfaces, proving the existence of the family amounts to showing that there is a primitive embedding $\mathcal{L}_G$ in $\Lambda_{K3}=U^{\oplus 3}\oplus E_8(-1)^{\oplus 2}$. By [2, propositions 6.2 and 7.7], there is a family of elliptic $K3$ surfaces whose Néron–Severi group is $U\oplus \Omega_G$. Now, any embedding of U in an even lattice M induces a splitting $M=U\oplus U^\perp$; in particular, in our case, we have $\Omega_G\subset U^{\oplus 2}\oplus E_8(-1)^{\oplus 2}\subset \Lambda_{K3}$. We fix a canonical basis $\lbrace e,f \rbrace$ for the distinguished copy of U and we consider the lattice $\mathbb{Z}(e+5f)\oplus \Omega_G$, which is a copy of $\mathcal{L}_G$ and is primitive in $\Lambda_{K3}$, since the two summands lies in two different direct summands of the big lattice. This give us a maximal family of $K3$ surfaces with a symplectic action of G.

For the very general element (S, H) in the family, the $\langle 10 \rangle$-polarization has divisibility 10 in the Néron–Severi group; in particular, (S, H) is Brill–Noether general and strongly smooth by theorem 3.3. Since these properties fail on a finite number of divisors in the moduli space, this means that also the general element in the family enjoys them (these families have positive dimension). For the same lattice-theoretic reasons, (S, H) will not lie in $D_{h,0}$ for $h=4,5$ nor $D_{1,-2}$, so its associated double EPW sextic is smooth and carries a symplectic action of G fixing the polarization D_A, by propositions 4.1 and 2.10.

Proposition 5.5 ensures that the group of automorphisms fixing D_A on the associated EPW sextic is isomorphic to $\operatorname{Aut}(S,\mathbb{P}(W))$ for a general (S, H), since in that case the projective model does not contain any line or conic.

5. Bounds on automorphisms of GM varieties

5.1. Bounds for Brill–Noether general K3 surfaces

In the next two sections, we provide some bounds on the automorphism group of GM varieties. Grassmannian hulls will play a fundamental role and the key point is the following observation.

Lemma 5.1.

Let Z be a strongly smooth GM variety with Grassmannian hull M_Z. Then $\operatorname{Aut}(Z,\mathbb{P}(W))\subset \operatorname{Aut}(M_Z)$.

Proof. Since S spans $\mathbb{P}(W)$, the restriction $\rho\colon\lbrace \alpha \in \operatorname{Aut}(\mathbb{P}(W)) | \alpha(S)=S\rbrace\to \operatorname{Aut}(S,\mathbb{P}(W))$ is an isomorphism of groups. By [22, corollary 2.11], any $\alpha\in \operatorname{Aut}(S,\mathbb{P}(W))$ is induced by an automorphism $(g_W,g_{V},g_L)$ of ordinary GM data, where the isomorphism $[g_W]\in \operatorname{PGL}(W\otimes L)=\operatorname{PGL}(W)$ is the restriction of $\left[\bigwedge^2 \left({g_V}_{|_{V_5}}\right)\right]$ to $\mathbb{P}(W)$. Hence ρ factors as $\lbrace \alpha \in \operatorname{Aut}(\mathbb{P}(W)) | \alpha(S)=S\rbrace\xrightarrow{\rho_1} \operatorname{Aut}(M_S) \xrightarrow{\rho_2} \operatorname{Aut}(S,\mathbb{P}(W))$. Since ρ is injective, the same holds for ρ ₁.

Let $S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be a Brill–Noether general $K3$ surface. The automorphism group $\operatorname{Aut}(S,\mathbb{P}(W))$ is the group of the automorphisms of the abstract surface S fixing H; this group is finite, see [6, chapter 5, proposition 3.3].

O’Grady [19] studied in detail the six-dimensional complete linear system $|H_2-2\delta|$. This linear system is naturally isomorphic to the space of quadrics in $\mathbb{P}(W)$ containing S and the associated map admits a very explicit description ([19, (4.2.5)]),

(12)\begin{align} \phi\colon S^{[2]} & \dashrightarrow |H_2-2\delta|^\vee\cong \mathbb{P}^5 \end{align}

(13)\begin{align} Z & \mapsto \lbrace \mathcal{Q}\in |V_6| \mbox{such that } \langle Z \rangle\subset \mathcal{Q} \rbrace . \end{align}

The map ϕ factors as a small contraction $c\colon S^{[2]}\to X_{A(S)^\perp}$ and $f\colon X_{A(S)^\perp}\dashrightarrow \mathbb{P}^5$. The latter can be identified to the double cover $f_{A(S)\perp}$ up to a finite number of flops [19, theorem 4.15]; in particular, it is generically 2-to-1 and induces a birational involution ι on $S^{[2]}$ which acts as minus the reflection with respect to the class $H_2-2\delta$ in cohomology [19, proposition 4.20]. The birational transformation c induces an identification of $\operatorname{Aut}_{D_{A(S)^\perp}}(X_{A(S)^\perp})$ with $\operatorname{Bir}_{H_2-2\delta}(S^{[2]})$, the group of birational endomorphisms fixing $H_2-2\delta$ and (5) translates as

\begin{equation*} 1\to \lbrace \operatorname{id},\iota \rbrace\to \operatorname{Bir}_{H_2-2\delta}(S^{[2]}) \xrightarrow{\lambda} \operatorname{Aut}(Y_{A(S)^\perp}) \to 1. \end{equation*}

We denote by $\mathfrak{G}_n$ the symmetric group on n objects and by $\mathfrak{A}_n$ the alternating group on n objects.

Proposition 5.2.

If S is strongly smooth, $\operatorname{Aut}(S,\mathbb{P}(W))$ acts symplectically on S and is isomorphic to one of the following groups:

(14)\begin{equation} \begin{gathered} \mathbb{Z}/n\mathbb{Z} \,\mbox{for }n= 1,2,3,4, \quad \quad D_n \,\mbox{for } n= 2,\ldots,6 \\ \mathfrak{A}_4, \quad \quad \quad \mathfrak{G}_4,\quad \quad \quad \mathfrak{A}_5.\quad \quad \end{gathered} \end{equation}

Proof. We prove simplecticity. We know that (5) splits, identifying the lifting of $\operatorname{Aut}(Y_{A(S)^\perp})$, which we denote by $\mathcal{S}$, with the group of automorphisms acting trivially on $H^{0,2}(X_{A(S)^\perp})$. Since c is an isomorphism in codimension one, $\mathcal{S}$ seen as a subgroup of $\operatorname{Bir}_{H_2-2\delta}(S^{[2]})$ acts symplectically on $S^{[2]}$. Inside $\mathcal{S}$ there is G, the lifting of $\operatorname{Aut}(S,\mathbb{P}(W))\subset \operatorname{Aut}(Y_{A(S)^\perp})$. We denote by $G^{\prime}\subset \operatorname{Bir}_{H_2-2\delta}(S^{[2]})$ the group of natural automorphisms $\alpha^{[2]}$ on $S^{[2]}$ induced by $\alpha\in \operatorname{Aut}(S,\mathbb{P}(W))$: if we can prove $G=G^{\prime}$ we are done, since $\alpha^{[2]}$ is symplectic if and only if α is.

Fix some $\alpha\in \operatorname{Aut}(S,\mathbb{P}(W))\subset \operatorname{Aut}(Y_{A(S)^\perp})$. A straightforward computation, using (12) and (7), gives $\phi\circ\alpha^{[2]}=\alpha\circ \phi$, hence $\lambda(G)=\lambda(G^{\prime})$. For $\tilde{\alpha}\in G$ the lifting of α, we have then $\tilde{\alpha}=\alpha^{[2]}$ or $\tilde{\alpha}=\alpha^{[2]}\circ \iota$. Note that $\alpha^{[2]}$ fixes the class H ₂, while ι only fixes $H_2-2\delta$ and its multiples [15, proposition 4.21 (b)], so the first equality holds if and only if $\tilde{\alpha}$ fixes H ₂.

We turn to $|H_2|$, as a linear system on $X_{A(S)^\perp}$, via the small contraction c; it is the pullback via the Hilbert–Chow contraction of $H^{\otimes 2}$ on the symmetric product $S^{(2)}$,

\begin{equation*} H^0(S^{[2]},H_2)=H^0(S^{(2)},H^{\otimes 2})=\operatorname{Sym}^2 W^\vee.\end{equation*}

The action of the lifting $\tilde{\alpha}$ of α to $X_{A(S)^\perp}$ is induced by the action of $SL(V_6)$ [25, proof of proposition A.2]; thus, $\tilde{\alpha}$ acts on the global sections of H ₂ since it induces a linear automorphism of W, see Eq. (7). So $\tilde{\alpha}^*H_2=H_2$, which provides us $\tilde{\alpha}=\alpha^{[2]}$ as desired.

To obtain the bound we observe that, by lemma 5.1, G is a finite subgroup of $\operatorname{Aut}(M_S)$ and the latter is isomorphic to $\operatorname{PGL}(2,\mathbb{C})$ [14, theorem 7.5], since the strongly smoothness of S implies the condition of generality for M_S in [14], cf. [22, proposition 2.22]. So G is either cyclic, dihedral, $\mathfrak{A}_4$, $\mathfrak{G}_4$, or $\mathfrak{A}_5$. For cyclic groups, the order is at most 8 by [31, theorem 4.5] and actually when the order is $n=7,8$ the lattice $\Omega_G^\perp$ (same notation as in proposition 4.2) does not contain an element of square 10, as a computation modulo 2n shows, so we can rule them out. Moreover, if G contains an order $n=5,6$ element, then $D_n\subset G$ by [2, proposition 8.1]. Finally, $n\leq 6$ for D_n by [10].

Remark 5.3.

In proposition 4.2, we provide some families of $K3$ surfaces with prescribed $\operatorname{Aut}(S,\mathbb{P}(W))$. According to proposition 5.2, the possible cases left are $\operatorname{Aut}(S,\mathbb{P}(W))\in \lbrace D_3,\mathfrak{A}_4,\mathfrak{G}_4,\mathfrak{A}_5 \rbrace$.

When S contains no line, $\phi:S^{[2]}\dashrightarrow |H_2-2\delta|$ behaves particularly well. Let $\mathcal{C}$ be the set of smooth conics inside S: by [19, claim 4.19], $|H_2-2\delta|$ is base-point-free and

(15)\begin{equation} Y^3_{A(S)^\perp}=\bigcup_{C\in \mathcal{C}} \phi(C^{(2)}). \end{equation}

When moreover S is strongly smooth, we are able to provide a bound for the automorphism group of the associated EPW sextic $Y_{A(S)}$, which is particularly useful when there is no conic on S; we used proposition 5.5 under this condition in the proof of proposition 4.2, in the previous section.

Definition 5.4.

We denote by $\operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W))\leq \operatorname{Aut}(S,\mathbb{P}(W))$ the subgroup of automorphisms acting trivially on $\mathcal{C}$.

Clearly $\operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W))$ appears in (14).

Proposition 5.5.

Let $S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be a strongly smooth Brill–Noether general $K3$ surface with N smooth conics on it. Suppose that S contains no line: the automorphism group of $Y_{A(S)}$ sits in an exact sequence

(16)\begin{equation} 1\to \operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W)) \to \operatorname{Aut}(Y_{A(S)}) \xrightarrow{g} \mathfrak{G}_{N+1} \end{equation}

where g sends $\alpha\in \operatorname{Aut}(Y_{A(S)})$ to its action on $Y^3_{{A(S)}^\perp}$, whose cardinality is N + 1.

Proof. The morphism g in (16) is well-defined by corollary 2.6 and its kernel is the subgroup of automorphisms of Y_A fixing pointwise $Y^3_{A^\perp}$. We are only left to prove that $\operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W))=\ker(g)$, we actually prove a bit more. Indeed, by (15), the fibres over $Y^3_{A(S)^\perp}$ are given by P_S and $C^{(2)}$ for $C\in \mathcal{C}$; by the same computations of proposition 5.2, the action of α on $\mathcal{C}$ induces a permutation on $\lbrace P_S,C^{(2)}_1, \ldots , C^{(2)}_N \rbrace$ hence on $Y^3_{A(S)^\perp}$, which is the same as the one induced by α as an element of $\operatorname{Aut}(Y_{A(S)})$ (note that, since α acts on M_S too by lemma 5.1, $\alpha^{[2]}$ always fixes P_S).

Remark 5.6.

The morphism g in (16) is not necessarily surjective. Indeed, let $(S,H)\in\mathcal{K}_{10}$ be a very general element of $D_{2,-2}$ (see definition 3.1). Then S contains exactly one smooth conic, say C. An involution of $Y_{A(S)}$ exchanging the two elements inside $Y^3_{A(S)^\perp}$ would lift to a symplectic involution on $X_{A(S)}$ (which is smooth for (S, H) general, see theorem 3.3), but then $X_{A(S)}$ would have Picard rank at least 9 by [12, corollary 5.2]. This is impossible since $X_{A(S)}$ and $S^{[2]}$ have the same Picard rank, cf. lemma A.4.

Furthermore, there exists $(S^{\prime},H^{\prime})\in D_{2,-2}$ such that $(S^{[2]}, H_2-2\delta)$ and $((S^{\prime})^{[2]},$ $H^{\prime}_2-2\delta^{\prime})$ are birational as varieties with a big and nef line bundle or equivalently $A(S)=A(S^{\prime})$ and $P_{S^{\prime}}$ is sent to $\phi(C^{(2)})$ via the map associated with $|H^{\prime}-2\delta^{\prime}|$. The argument above proves that (S, H) and $(S^{\prime},H^{\prime})$ are not isomorphic.

5.2. Bounds in greater dimension

In §5.1, we dealt with two-dimensional GM varieties. Now we consider GM varieties of dimension 3 and 4. By [22, proposition 3.21], their automorphism group is finite and trivial in the general case.

Remark 5.7.

By proposition 4.1, whenever the associated Lagrangian A(Z) does not lie in $\Delta\cap \Pi$, automorphisms of prime order of Z have order at most 11, because the automorphism lifts to $X_{A(Z)}$ and $X_{A(Z)^\perp}$, of which at least one is smooth, and a symplectic automorphism of prime order on a deformation of a Hilbert scheme of two points has order at most 11, see [11, corollary 2.13].

Proposition 5.8.

Let Z be a smooth ordinary GM variety of dimension 3 and $G\subset \operatorname{Aut}(Z)$. Then there exists a group H among

(17)\begin{equation} 1, \quad \quad A_4, \quad \quad \mathfrak{G}_4, \quad \quad A_5, \quad \quad \mathbb{Z}/k\mathbb{Z} \quad\mbox{or} \quad D_k \quad\mbox{for }k\geq 2 \end{equation}

and $r\geq 1$ such that G sits in an exact sequence $1 \to \mathbb{Z}/r\mathbb{Z} \to G \to H \to 1 $.

Proof. The Grassmannian hull M_Z is smooth since Z is three-dimensional and $\operatorname{Aut}(Z,\mathbb{P}(W))=\operatorname{Aut}(Z)$ since Z has Picard rank one. By [14, theorem 6.6], there is a short exact sequence

(18)\begin{equation} 1\to \mathbb{C}^4 \rtimes \mathbb{C}^* \to \operatorname{Aut}(M_Z)\xrightarrow{\pi} \operatorname{PGL}(2,\mathbb{C}) \to 1 , \end{equation}

thus G is an extension of $H=\pi(G)$ by $N=G\cap (\mathbb{C}^4 \rtimes \mathbb{C}^*)$. The group G is finite by lemma 5.1, so $H\subset \operatorname{PGL}(2,\mathbb{C})$ has to appear in (17). As for N, it is finite inside $\mathbb{C}^4 \rtimes \mathbb{C}^*$, so it is isomorphic to a finite subgroup of $\mathbb{C}^*$.

Remark 5.9.

In [29], Debarre and Mongardi produced an explicit example of EPW sextic with automorphism group isomorphic to $\operatorname{PSL}(2,\mathbb{F}_{11})$. This allows them to find examples of smooth ordinary GM threefolds with automorphism group isomorphic to $\mathbb{Z}/11\mathbb{Z}$, D ₆, $\mathbb{Z}/6\mathbb{Z}$, $\mathbb{Z}/3\mathbb{Z}$, D ₅, D ₂, $\mathbb{Z}/2\mathbb{Z}$.

The spin group $\operatorname{Spin}(n)$, for $n\geq 2$, is the universal cover of the special orthogonal group $\operatorname{SO}(n)$. It is possible to study finite subgroups of $\operatorname{Spin}(5)$ using this definition: finite subgroups of $\operatorname{SO}(5)$ have been classified, see [21, corollary 2].

The quotient by $\lbrace \pm 1\rbrace$ in the result below comes from the fact that, for any $$\alpha\in \operatorname{PGL}(V_5)$$ acting on the Grassmannian hull M_Z, a representative in $$\operatorname{GL}(V_5)$$ of the form $$\begin{pmatrix} \bar{T} & \underline{0}\\ {^t\underline{b}} & b \end{pmatrix}$$ is chosen, such that $$\operatorname{det}(\bar{T})=1$$. A representative in this form for α is unique, up to multiplication by −1. See [14, proposition 5.2] for details.

Proposition 5.10.

Let Z be a smooth ordinary GM variety of dimension 4 and $G\subset \operatorname{Aut}(Z)$. Then G is the quotient by $\lbrace \pm 1\rbrace$ of a group $\tilde{G}$ such that there is an exact sequence

\begin{equation*} 1\to \mathbb{Z}/r\mathbb{Z} \to \tilde{G} \to H \to 1 \end{equation*}

for some $r\geq 1$ and H a finite subgroup inside $\operatorname{Spin}(5)$.

Proof. As in proposition 5.8, we just have to impose finiteness for a subgroup of $\operatorname{Aut}(M_Z)$. By [14, proposition 5.2], the group G is an extension of a subgroup of $\left( \operatorname{Sp}(4,\mathbb{C})\times \mathbb{C}^* \right) /\lbrace \pm 1\rbrace$ by some $N\subset \mathbb{C}^4$. Actually $N=\lbrace 1 \rbrace$ as G is finite, hence G is isomorphic to the quotient of a finite subgroup of $\operatorname{Sp}(4,\mathbb{C})\times \mathbb{C}^*$. Any finite subgroup of $\operatorname{Sp}(4,\mathbb{C})$ lies inside its maximal compact subgroup, which is the compact symplectic group $\operatorname{Sp}(2)$. In turn, $\operatorname{Sp}(2)$ is isomorphic to $\operatorname{Spin}(5)$.

Remark 5.11.

In [29], Debarre and Mongardi produced smooth ordinary GM fourfolds with automorphism group isomorphic to D ₆, $\mathbb{Z}/3\mathbb{Z}$, D ₅, $\mathbb{Z}/5\mathbb{Z}$, D ₂, $\mathbb{Z}/2\mathbb{Z}$.

Corollary 5.12.

Consider $A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)} - \Sigma$ and $G\subseteq \operatorname{Aut}(Y_A)$. Suppose that G fixes a point $[v]\in Y^k_A$ or a point $[V_5]\in Y^k_{A^\perp}$ for $k\in \lbrace1,2\rbrace$.

1. (1) If k = 2, then G sits in an exact sequence $1 \to \mathbb{Z}/r\mathbb{Z} \to G \to H \to 1 $ for some $r\geq 1$ and H a group in (5.8).

2. (2) If k = 1, then G is the quotient by $\lbrace \pm 1\rbrace$ of a group $\tilde{G}$, where $\tilde{G}$ is an extension of some finite subgroup of $\operatorname{Spin}(5)$ by a cyclic group $\mathbb{Z}/r\mathbb{Z}$.

Proof. We consider the case of G fixing $[V_5]\in Y^k_{A^\perp}$, the dual case follows by corollary 2.6. We denote by Z the ordinary GM variety associated with the Lagrangian data $(V_6,V_5,A)$: by theorem 2.17, it is a strongly smooth GM variety of dimension $5-k$. By proposition 4.1, there is an inclusion $G\subset \operatorname{Aut}(Z,\mathbb{P}(W))$, then the result follows from propositions 5.8 and 5.10.

If the associated Lagrangian A(Z) lies in Δ or Σ we can be more precise.

Definition 5.13.

A plane $\pi\subset G(2,V_5)$ is a σ-plane if it is of the form $\mathbb{P}(v\wedge V_4)$ for some four-dimensional vector space $V_4\subset V_5$ and $v\in V_4-\lbrace 0\rbrace$. A τ-quadric surface in $G(2,V_5)$ is a linear section of $G(2,V_4)$ for some four-dimensional $V_4\subset V_5$.

Corollary 5.14.

Let Z be a smooth ordinary GM variety of dimension 4 and associated Lagrangian subspace A(Z). If one of the following holds:

• • Z contains a σ-plane,

• • Z contains a τ-quadric surface,

then there is an exact sequence $1\to N \to \operatorname{Aut}(Z) \to \mathfrak{G}_{n} $, where $n=|Y^3_{A(Z)}|$ (resp. $n=|Y^3_{A(Z)^\perp}|$) if Z contains a σ-plane (resp. a τ-quadric surface), and N is one of the groups in the list at (14).

Proof. By [24, theorem 4.3, Item c] for σ-planes, [24, remark 5.29] and [27, Section 7.3] for τ-quadric surfaces, the associated Lagrangian subspace A(Z) lies in Δ or Π. So there is a strongly smooth Brill–Noether general $K3$ surface whose associated Lagrangian subspace is A(Z). Then we use proposition 5.5.

The coarse moduli space of smooth GM varieties of dimension 4 is constructed in [26]. Inside it, the family of ordinary GM varieties containing a σ-plane (resp. a τ-quadric surface) has codimension 2 (resp. 1), see [24, remark 5.29]; the general members of both families are rational, see [27, propositions 7.1 and 7.4].