Structured Replacement Policies for Offshore Wind Turbines

Abstract

We consider the problem of optimally maintaining an offshore wind farm in which major components progressively degrade over time due to normal usage and exposure to a randomly varying environment. The turbines exhibit both economic and stochastic dependence due to shared maintenance setup costs and their common environment. Our aim is to identify optimal replacement policies that minimize the expected total discounted setup, replacement, and lost power production costs over an infinite horizon. The problem is formulated using a Markov decision process (MDP) model from which we establish monotonicity of the cost function jointly in the degradation level and environment state and characterize the structure of the optimal replacement policy. For the special case of a two-turbine farm, we prove that the replacement threshold of one turbine depends not only on its own state of degradation but also on the state of degradation of the other turbine in the farm. This result yields a complete characterization of the replacement policy of both turbines by a monotone curve. The policies characterized herein can be used to optimally prescribe timely replacements of major components and suggest when it is most beneficial to share costly maintenance resources.

1. Introduction

Global demand for clean and renewable energy sources, such as wind and solar power, is motivated by growing concerns about environmental sustainability and climate change. Like many other industrialized nations, the United States has committed to reducing harmful carbon emissions while increasing the proportion of power generated by non-fossil-based sources over the next few decades. Specifically, it plans to (1) cut carbon emissions in half by 2030, (2) reach 100% clean electricity by 2035, and (3) achieve net-zero carbon emissions by 2050 [20]. However, these ambitious goals can only be attained if the supply and reliability of renewable energy sources are sufficient to meet consumer demand while remaining economically competitive with fossil fuel sources. Some in the operations community have considered optimal investment strategies for renewable energy technologies, such as wind and solar [3, 30]. Wind energy is playing an increasingly important role in the move towards more sustainable forms of energy because it is clean, accessible, affordable, and inexhaustible. Wind energy is produced by a wind turbine—a system designed to convert kinetic energy (wind) into mechanical energy to generate electricity. Wind turbines are typically comprises a structural tower, the rotor and hub (including three turbine blades), and the nacelle, which houses most of the mechanical and electrical systems, including the gear box and generator, among others. A collection of wind turbines operating in proximity to one another is known as a wind farm.

Although wind farms can be land-based (onshore) or sea-based (offshore), the vast majority of wind generation in the USA originates from onshore systems [47]. New onshore wind farm installations are very costly, as they require significant capital investments, supporting infrastructure and connection services, and these costs are substantially higher for offshore wind farms [52]. However, due to escalating demand for wind energy and depletion of available land resources, offshore wind has received a great deal of attention over the past two decades. Though it is more costly, offshore wind holds a few advantages over its onshore counterpart. First, stronger and more consistent wind resources are available offshore, thereby increasing generating capacity and improving reliability. Second, offshore wind farms are usually located in vast open water areas, allowing for large-scale installations. Finally, because offshore wind farms are distant from dense population centers, their negative visual and auditory impacts are significantly reduced. Despite these advantages, wide-scale adoption of offshore wind has not yet been realized due to significant capital investment costs and its levelized cost of energy (LCOE)—an aggregate economic measure that is used to compare lifetime power generation costs associated with different technologies [65]. The three components of this aggregate cost are broadly categorized as capital expenditures, operational expenditures, including operations and maintenance (O&M) costs, and decommissioning expenditures. O&M costs typically account for 20–50% and 35–50% of the cost of producing onshore and offshore wind energy, respectively [67, 74]. Moreover, the replacement of large, critical components such as turbine blades, gear boxes, and generators constitutes as much as 75% of total O&M costs [40, 70]. Therefore, the viability of large-scale offshore wind generation hinges on a reduction of O&M costs and, ultimately, the LCOE.

While similarities exist between the maintenance of onshore and offshore wind farms, offshore farms are especially difficult to maintain due to accessibility issues, environmental conditions (e.g., wind speed and wave height), safety concerns, and the need for costly, specialized crews and equipment to perform operations. Accessibility is usually associated with their distance from the shore, sea conditions, and availability of sea vessels. When unanticipated failures occur, helicopters are often deployed to reduce delays in accessing the turbines. Another unique feature of offshore wind farm maintenance is the need for different types of vessels to carry crews, replacement parts, and equipment. In addition to crew transfer vessels (CTVs) and service operational vessels (SOVs), a jack-up vessel (JUV) is essential for replacement of large, critical components such as turbine blades and generators [70]. Because it may be cost-prohibitive for a wind farm operator to own such resources, they must be chartered from external providers. Dalgic et al. [15–17] have documented several complicating factors associated with the use of JUVs. The accessibility and availability issues associated with offshore wind farm maintenance are aptly detailed in [25, 52, 57, 58, 61, 62].

These considerations motivate our work here in which we seek to prescribe optimal replacement policies for offshore wind farms that minimize the long-run expected total discounted setup, replacement, and lost production costs. Critical to our approach is the inclusion of both economic and stochastic dependence between the turbines in a wind farm. Economic dependence is present because significant setup costs are incurred any time replacements are performed. These costs can be shared among multiple turbines, allowing operators to exploit economies of scale. Stochastic dependence refers to the association of the turbines’ degradation processes stemming from their common operation in a randomly varying, exogenous environment. The repair or replacement of turbine components can be either preventive (occurring before a failure) or reactive (occurring after a failure); in either case, a per-replacement cost is incurred. Substantial downtime costs (in the form of lost revenue) are also incurred whenever a major component fails and/or a turbine is taken offline to perform replacements. We formulate and analyze a Markov decision process (MDP) model to prescribe optimal replacement policies and provide practical insights to wind farm operators.

1.1. Relevant literature

An offshore wind farm can be viewed as a collection of complex, progressively degrading multiunit systems operating in a dynamic environment. A multiunit system can refer either to one system with multiple (dependent or independent) components or a collection of systems that are dependent in some way; we adopt the latter view for offshore wind farms. Many other researchers embrace the first view in order to devise tractable maintenance policies for multiple independent units. For instance, Tian and Liao [68] examined policies for general multiunit systems with several identical, independent units and numerically obtained dual threshold-type policies. Zhang et al. [80] decomposed a multiunit system and established an opportunistic dual-threshold policy for the maintenance of each single-unit system. Arts and Basten [7] proposed a condition-based maintenance (CBM) model for each component of a two-component system independently by considering both scheduled and unscheduled downtime. After prescribing optimal maintenance policies for each component, they synchronized scheduled downtime in order to maintain both units simultaneously. Such decomposition procedures usually lead to tractable maintenance policies; however, it is well-known that interdependent, degrading systems present significant challenges to strategic maintenance planning [19, 49, 73, 83], especially when multiple types of dependence exist. Olde Keizer et al. [50] surveyed several dependence types prevalent in CBM strategies for degrading systems. Here, we focus primarily on economic and stochastic dependence.

First, economic dependence exists in multiunit systems generally, and in offshore wind farms specifically, because maintenance operations are typically performed jointly on several units to share setup costs and reduce the total downtime overall. Ko and Byon [34] considered the problem of maintaining a large number of homogeneous units that degrade independently over time but are economically dependent due to shared setup costs. Salari and Makis [59] examined economic dependence between the units of a production system in which the production rate of each unit decreases as the degradation level increases. Zhu et al. [81] considered a multicomponent system with heterogeneous, independently degrading units and significant shared maintenance setup costs. Zhu et al. [82, 83] formulated a multistage stochastic integer program to optimize the maintenance of a multicomponent system subject to economic dependence. They derived structural results for a simpler two-stage version of the model to enhance tractability. Related to economic dependence is resource-based dependence, in which units are linked through a shared and limited set of resources [1, 36, 63]. Olde Keizer et al. [51] used a MDP model to jointly optimize maintenance activities and spare parts inventory of a multicomponent system in which the components share identical spares. Maintenance is feasible only if spares are available, thereby inducing resource-based dependence. They obtained optimal replacement and ordering policies that minimize the total long-run average cost per time unit. Similarly, de Pater and Mitici [18] considered multiple repairable components whose maintenance activities are linked via the availability of spares. They integrated remaining useful life (RUL) estimation with a limited inventory of spares to devise a predictive maintenance model subject to resource-based dependence. Wind turbines may be forced to sit idle awaiting the resources needed to restore them to working condition, resulting in economic losses in the form of lost revenues.

Second, stochastic dependence can exist between the units in a multiunit system in at least three ways: via load sharing, dependence of degradation processes due to structural dependencies or the association of degradation processes via a common operating environment. The first type exists when the failure or degradation of one unit influences the lifetimes of other units. For example, if additional power production is required of one wind turbine due to the failure of another, load sharing can lead to accelerated degradation and premature failure of the operational unit [9, 69, 76, 78, 80]. Structural dependence refers to the physical configuration of components within the system, for example, the components are connected in series, parallel, or some other manner [5, 38, 75, 79]. Third, stochastic dependence can be induced by exogenous factors, such as environmental conditions, that simultaneously influence the degradation processes of individual units. Reliability and availability indices, as well as optimal maintenance strategies, for single-unit systems operating in a randomly evolving environment have been well studied [31–33, 35, 77]. Random shock processes have often been used to model the operating environment and its impact on competing degradation processes [14, 28, 55, 56, 75]. Of particular interest to our work here is a model by Ulukuş et al. [71], who examined the optimal replacement of a single-unit system subject to environment-driven degradation. Using an MDP model, they established the existence of an optimal degradation-based threshold replacement policy for each environment state. Furthermore, they conjectured conditions for which the thresholds are monotone; however, this conjecture was based only on empirical evidence. Abdul-Malak et al. [2] extended the model in [71] to the case of multiple units operating in a shared environment, where stochastic dependence stems from the association of degradation processes via the common environment. They established monotonicity of the value function over the entire state space, although they did not characterize the optimal policy. Instead, numerical approximations were used to illustrate the replacement policies, and for the special case of a single unit, they provided a partial resolution to the conjecture in [71] and proved conditions needed to ensure that the replacement thresholds are monotone (for each environment state). Our work differs from prior models in that we seek to establish structured replacement policies that consider the following: (1) economic dependence due to shared setup costs, (2) stochastic dependence induced by a common, randomly evolving environment, and (3) the impact of production losses that also depend on the environment state.

1.2. Summary of contributions and results

Our main contributions and results are summarized as follows:

1. (1) The problem we study has a unique feature: The turbines in the farm are exposed to a common environment that evolves randomly over time. In particular, the degradation processes of the turbines are associated with each other through their common exposure to the environment. Therefore, in addition to economic dependence induced by the sharing of setup costs, there is stochastic dependence among the turbines. We create and analyze an MDP model that incorporates economic and stochastic dependence, both of which significantly influence maintenance strategies.

2. (2) We establish structural properties that lead to a characterization of the optimal replacement policy. First, we show that insights from the analysis of single-unit systems carry over to our setting. Specifically, for a single turbine, the optimal policy is monotone in its degradation level when the degradation levels of all other turbines are fixed, as seen in Figure 1(a). For a fixed turbine 1 degradation level (level A), there is a turbine 2 degradation threshold (level B) below which it is not optimal to replace and above which it is optimal to replace. Other models assume that it is always optimal to replace a failed system, implying that a replacement threshold always exists. However, in our setting, it is possible that replacement of a failed turbine is not prescribed due to significant shared setup costs. Second, for the special case of two turbines, we provide a novel result that, for a given level of degradation of one turbine, the optimal policy is monotone in the degradation level of the other turbine beyond a certain point. Referring to Figure 1(b), for a fixed turbine 2 degradation level (level D), beyond the point X, there is a threshold degradation level of turbine 1 (level C) below which it is not optimal to replace turbine 2, but above which it is optimal. Similar to the prior case, it may not be optimal to replace turbine 2 when turbine 1 is failed. Third, we show that these thresholds (if they exist) are monotone in the degradation level of turbine 2. That is, in Figure 1(c), when the degradation level of turbine 2 is level G (level H), the optimal threshold for replacing turbine 2 is level F (level E) over the degradation level of turbine 1. Our results show that level E is necessarily less than or equal to level F. As a byproduct, the region where the optimal policy replaces both turbines (the opportunistic region) is completely characterized by a monotone curve (depicted by the dotted curve in Figure 6) in the space of degradation levels of the two turbines.

3. (3) Our numerical results provide a procedure to assess the value of incorporating dependence and adequately modeling the environment, as well as the consequences of incorrect assumptions on the replacement policies and costs. Our results provide insights as to how the replacement policies depend on the environment and underlying assumptions.

The remainder of the paper is organized as follows. In Section 2, we describe the degradation model and formulate the replacement problem using an MDP model. Section 3 establishes properties of the cost function and the structure of the optimal replacement policy. In Section 4, numerical examples are used to illustrate replacement policies and demonstrate the value of incorporating dependence and the environment. Finally, some concluding remarks are provided in Section 5.

Figure 1. Visualization of the optimal replacement policy of turbine 2. (a) Monotone policy in its own degradation level. (b) Monotone policy in the other turbine’s degradation level. (c) Monotonicity of thresholds.

2. Model and problem formulation

In this section, we first provide a mathematical model of wind turbine degradation due to normal operation and the influence of a randomly evolving environment. Subsequently, we formulate a stochastic optimization problem to minimize the expected total discounted setup, replacement, and downtime costs over an infinite horizon.

2.1. Degradation model

Consider a finite collection $\mathcal{N}:=\{1,\ldots,N\}$ of N wind turbines operating in an offshore wind farm. Critical wind turbine components (e.g., generators, transformers, rotors, and turbine blades) degrade over time due to normal usage and the influence of their ambient operating environment. For example, it is well known that ocean air is corrosive to wind turbine blades due to its higher salt content [22]. In what follows, the phrase “turbine degradation” refers to the degradation of a single, critical component whose replacement requires substantial time, effort, and resources. That is, we do not specify which component but refer more generally to the turbine’s degradation. Each turbine degrades until its cumulative level of degradation first reaches, or exceeds, a critical deterministic threshold ξ (ξ > 0), at which time the turbine is said to be failed. In the problem formulation that follows, we consider discrete decision epochs $m\in \mathbb{Z}_+:=\{0,1,2,\ldots\}$ and define a period as the time between two consecutive decision epochs. Let $X_n(m)$ denote the degradation of turbine $n\in\mathcal{N}$ at decision epoch m with state space $[0,\xi]$. Here, level 0 means a turbine is in as-good-as-new condition. Next, define the vector-valued process $\mathbb{X}=\{\left(X_1(m),\ldots,X_N(m)\right):m\in \mathbb{Z}_+\}$ describing the joint degradation process of all N turbines with continuous state space $\mathcal{X}:=[0,\xi]^N$. At decision epoch m, turbine n is said to be working if $X_n(m)\in [0,\xi)$ and failed if $X_n(m)\ge\xi$. Hereafter, and without loss of generality, we set the failure threshold ξ = 1. Stochastic processes are used extensively in the reliability and maintenance optimization communities to model the temporal and/or spatial evolution of degradation in systems. For a single system, $\{X_n(m):m\in \mathbb{Z}_+\}$ is often assumed to evolve as a discrete-time Markov chain (DTMC) on a finite state space, or when a continuous-time, continuous-state process is more appropriate, degradation can be modeled using gamma, Wiener, or other Lévy processes [6, 10, 38, 48].

For our purposes here, it is appropriate to model not only degradation but also the influence of a randomly varying environment on the evolution of degradation. Wind turbines are often located in proximity to one another within a wind farm; thus, they experience not only similar environmental conditions but also wake effects resulting from changes in wind speed [21]. Although the turbines operate independently, their degradation processes exhibit stochastic dependence due to their exposure to a common environment. Monbet et al. [44] surveyed stochastic models—many of which are Markovian—that can be used to model wind conditions and the sea state from time series data. Although the environment realistically evolves continuously in time and space, researchers have assumed that it evolves over finitely many discrete states. Representative examples include [2, 11, 31, 32, 71]. In those specific models, each environment state maps to a unique, positive degradation rate. That is, if the environment occupies state i, then the system degrades linearly at a rate $r(i) \gt 0$. Consequently, in the absence of any type of maintenance intervention, the degradation process evolves as a piecewise linear function of time (almost surely). In contrast to those models, our main theoretical results do not impose such restrictive assumptions but only require that future degradation can be predicted using the current level of degradation and the environment state. The discrete environment states are essentially aggregations of several factors, including wind speed, ambient temperature, relative humidity, wave height, and others. Hence, these states can be viewed as approximations of the true (continuous) environment states over time. For example, if the environment evolves as a diffusion model with a randomly evolving drift coefficient, the distinct environment states can be viewed as the average behavior of the diffusion process within an interval of time. Flory et al. [23] employed Markov chain Monte Carlo methods to model such behavior.

Let J(m) denote the state of the environment at decision epoch m and assume $\mathbb{J}:=\{J(m):m\in\mathbb{Z}_+\}$ evolves as a temporally homogeneous DTMC on the finite state space $\mathcal{L}:=\{1,\ldots,L\}$ with $L\in \mathbb{N}$. The environment states are ordered such that state 1 represents the most favorable condition for maintenance activities and state L represents the least favorable condition. To account for accessibility, resource availability, and maintenance feasibility issues, we can append $\mathcal{L}$ with additional states that reflect conditions under which maintenance cannot be performed (e.g., when wind speeds and/or wave heights are excessive, transport vessels are unavailable and/or crews are unavailable). In such cases, the only feasible action is “do nothing” and wait until the next decision epoch. The one-step transition probability matrix (TPM) of $\mathbb{J}$ is denoted by $\mathbf{P}=[p_{\ell k}]$, where $p_{\ell k}$ is the conditional probability that the environment state is k at decision epoch m + 1, given that it was $\ell$ at epoch m for any $m\in \mathbb{Z}_+$. The number of distinct environment states (L) can be estimated using the well-known Bayesian information criterion [23, 60]. Anastasiou and Tsekos [4] concluded that six-to-eight distinct states are sufficient to model environmental conditions using a first-order, stationary Markov chain and noted that the stationarity assumption has been employed to model wind speed and wave height in other maritime applications [8, 39]. It is well-known that weather conditions exhibit strong seasonality effects, as highlighted and modeled by Byon et al. [12, 13]. To account for these effects, one can employ a different DTMC model for different seasons of the year. The transition probabilities can be estimated using proportions data [29, 41] or by mining time series data [66]. A realization of the joint process $(\mathbb{X},\mathbb{J})$ is an $(N+1)$-dimensional vector $(\textbf{x},\ell)$, where $\textbf{x}=(x_1,\ldots,x_N)\in\mathcal{X}$ and $\ell\in \mathcal{L}$.

We assume that future degradation can be predicted via the current state $(\textbf{x},\ell)$. For each turbine $n\in \mathcal{N}$ and a triplet $(x_n,\ell,t)$, let $f_n:[0,1]\times \mathcal{L} \times \mathbb{R}_{+}\to \mathbb{R}_{+}$ be a generic mapping that predicts the degradation of turbine n after t time units when its current degradation level is x_n and the environment state is $\ell$. It can assume any number of functional forms (e.g., linear, piecewise linear, exponential, etc.); however, we impose no such assumption. This mapping satisfies the initial condition $f_n(x_n,\ell,0)=x_n$, and we impose the natural assumption that, for each $n\in \mathcal{N}$, f_n is monotone nondecreasing in $(x_n,\ell,t)$, i.e., $f_n(x_n,\ell,t)\le f_n(x^{\prime}_n,\ell^{\prime},t^{\prime})$ for any $x_n\le x^{\prime}_n$, $\ell\le \ell^{\prime}$, and $t\le t^{\prime}$. That is, future degradation is assumed to be higher than the present level of degradation, and because the state space $\mathcal{L}$ is ordered, higher environment states lead to accelerated degradation of the units; however, we impose no additional assumptions on f_n. Condition monitoring or sensor data (e.g., from a Supervisory Control and Data Acquisition (SCADA) system) can be used to estimate the evolution of degradation and/or the RUL of components using the techniques developed in [11, 24], for example. Papadopoulos et al. [53] suggest that when a dynamic environment is considered, the RUL can be estimated using the models developed by Bian et al. [11].

2.2. Problem formulation

Maples et al. [40] note that the majority of wind turbine O&M costs are due to the replacement of critical components (e.g., generators, transformers, rotors, and turbine blades). Thus, optimally timed replacements that exploit economies of scale can reduce O&M costs for wind farms and the overall cost of wind energy to consumers. Here, we present an MDP model to formulate the problem of optimally replacing wind turbines with the objective of minimizing the expected total discounted, setup, replacement and downtime costs over an infinite horizon.

Periodic inspections, which are assumed to be costless and instantaneous, are performed to reveal the state of each turbine perfectly (i.e., the state is completely observable). Such observations can be obtained, for example, by using sensors attached to critical components and tracking important signals of degradation, such as vibration or acoustic signals [11]. A decision to replace or not replace a unit is made by a decision maker (DM) just after an inspection such that $\mathbb{Z}_+=\{0,1,\ldots\}$ is the set of decision epochs, and the time between any two consecutive inspections is a period in the MDP model. The state of the system is the $(N+1)$-dimensional vector $(\textbf{x},\ell)$ whose state space is $\Gamma:=\mathcal{X} \times \mathcal{L}$. For any decision epoch $m\in \mathbb{Z}_+$, let $a_n\in \{0,1\}$ denote the action taken for turbine $n\in \mathcal{N}$, where $a_n=0$ means “do nothing” and $a_n=1$ means “replace.” Therefore, the N-dimensional vector $\textbf{a}=(a_1,\ldots,a_N)$ describes the actions taken by the DM, and the action space is $\mathcal{A}:=\{0,1\}^N$. Whenever replacement is prescribed, the turbine must be shut down for one full period. Replacement activities commence immediately after the inspection and end just prior to the subsequent inspection. We assume that a period is sufficiently long to ensure that replacements are completed during a single period. Because the average time to complete a major replacement is between 2 to 7 days (see Maples et al. [40]), the time between two consecutive decision epochs is 1 week in our model. However, it is important to note that the length of the period can be adjusted to account for different time scales when the model is used to prescribe the optimal timing of maintenance actions for minor components. In such cases, the period length can be set to 1 day. All replacements are assumed to be perfect and reset the degradation to level 0. For the remainder of the paper, vector inequalities are assumed to hold component-wise. That is, $\textbf{a}\le \textbf{a}^{\prime}$ implies $a_n\le a_n^{\prime}$, and $\textbf{x}\le \textbf{x}^{\prime}$ implies $x_n\le x_n^{\prime}$ for each $n\in \mathcal{N}$.

Let ${\hat{\textbf{x}}}_n(x_n,\ell,a_n)$ be the predicted level of degradation of turbine n at the start of the next period when the current degradation and environment states are x_n and $\ell$, respectively, and action a_n is taken. Then,

(1)\begin{equation} {\hat{\textbf{x}}}_n(x_n,\ell,a_n) = \begin{cases} \min\{1,f_n(x_n,\ell,1)\}, & \text{if}\ a_n=0,\\ 0, & \text{if}\ a_n=1. \end{cases} \end{equation}

Following a replacement, degradation is returned to level 0 (an as-good-as-new condition), and without intervention, the turbine either fails or assumes some higher degradation level $f_n(x_n,\ell,1)$.

Next, we describe the setup and other costs associated with replacements in an offshore wind farm. If at least one unit is replaced, a setup cost c_s ($c_s \gt 0$), which comprises significant equipment and crew transfer costs, is incurred. For offshore wind farms, the distance between the base station and the shoreline is usually significant such that helicopters and SOVs and/or JUVs must be used to transport needed equipment and crews to the farm. Additionally, heavy equipment is required to perform replacements; hence, the setup cost c_s can be substantial (in excess of £250,000 per day of operation; see [42, 64]). However, this cost can be shared if multiple turbines are replaced in the same period. The shared setup cost induces economic dependence between the turbines. Second, a per-replacement cost c_r ($c_r \gt 0$) is incurred for every turbine that is replaced. This cost compromises the procurement cost of a new component, as well as the labor hours needed to complete the replacement. Finally, a downtime cost is incurred for any turbine that either (1) fails in the time interval between two inspections or (2) is taken offline so that a replacement can be performed. This cost corresponds to lost power production due to turbine unavailability [40]. As noted by Wan et al. [72], power output is a function of wind speed and is typically characterized by a monotone increasing, S-shaped curve that is convex at low wind speeds, concave after an inflection point, and flat beyond an upper speed threshold [26]; therefore, we assume the downtime cost rate is a function of the environment state $\ell$. Let $c_d(\ell)$ denote the downtime cost per unit time per turbine when the environment is in state $\ell$. For example, if wind speed instantiates the environment, disjoint intervals of speed may correspond to different regions of the power output curve, as depicted in Figure 2(a). Three distinct wind speed ranges (low, medium, and high) result in distinct regions of power output.

Figure 2. Partition of wind speed into disjoint intervals with distinct downtime cost rates. (a) A typical wind power curve. (b) Downtime cost rate versus environment.

Figure 2(b) illustrates how the wind speed intervals can be mapped to environment states in order to associate a downtime cost rate with each state. Wind speed can be mapped to power production using the well-known method of bins, as prescribed by the International Electrotechnical Commission [27]. In this specific illustration with three environment states, we divide the wind speed range into three intervals of equal length, with the lowest environment state (state 1) corresponding to wind speeds that exceed the cutoff speed; therefore, we assign a positive downtime cost rate for this state. We assume the downtime cost rate $c_d(\ell)$ is strictly increasing in $\ell\in \mathcal{L}$ and bounded from above. That is, $0 \lt c_d(1) \lt \cdots \lt c_d(L) \lt \infty$.

We now describe the degradation dynamics of a particular turbine $n\in \mathcal{N}$ with and without replacement. Figure 3(a) and 3(b) depicts scenarios in which the environment state is fixed at $\ell$ throughout the period. From Figure 3(a), we see that at decision epoch m, the turbine can either be failed (point A) or working (point B).

Figure 3. Degradation evolution of a turbine within a period with and without replacement. (a) A turbine with replacement. (b) A turbine without replacement.

For either case, if replacement is prescribed at time m, the turbine is down for the entire period and is restored to a new condition at decision epoch m + 1 (point C); hence, the cost $c_d(\ell)+c_r$ is incurred in each of the two scenarios of Figure 3(a). Figure 3(b) depicts the same two scenarios in which the turbine is either failed (point A) or working (point B) at time m. Here, if replacement is not prescribed at decision epoch m and the unit is failed, it remains failed at m + 1 so that the incurred cost is $c_d(\ell)$. On the other hand, if the turbine is working at time m (point B) and replacement is not prescribed, two possible scenarios (represented by the dotted lines) may occur. First, the turbine continues degrading and reaches the critical threshold (ξ = 1) before decision epoch m + 1 (point D). In this case, the turbine is down from the crossing time (point D) until the start of the next period (point E). This downtime is denoted by $\tau_n(x_n,\ell)$ and is obtained via the predictive function $f_n(x_n,\ell,t)$. Specifically, let $\tau_n(x_n,\ell)$ denote the predicted within-period downtime if no replacement is performed ($a_n=0$), x_n is the degradation at the start of the period, and the environment state is $\ell$. Then,

(2)\begin{equation} \tau_n(x_n,\ell)=\max\left\{0,1-\inf\left\{t\ge 0: f_n(x_n,\ell,t)\ge 1\right\}\right\}. \end{equation}

For this first scenario, the downtime cost $c_d(\ell)\cdot \tau_n(x_n,\ell)$ is incurred for turbine n. For the second scenario, the turbine is working at time m (point B), but its level of degradation does not reach the failure threshold before decision epoch m + 1. Rather, it continues operating for the entire period and evolves to a higher level of degradation at decision epoch m + 1 (point F). In this case, $\tau_n(x_n,\ell)=0$ so that no downtime cost is incurred. Finally, we note that if the turbine is failed at decision epoch m (point A) and replacement is not prescribed, it remains failed at decision epoch m + 1 (point E). Since $f_n(1,\ell,t)=1$ for all $\ell\in \mathcal{L}$ and $t\ge 0$, it follows that $\tau_n(1,\ell)=1$. Therefore, the cost $c_d(\ell)\cdot 1=c_d(\ell)$ is incurred.

For notational convenience, define an indicator function $\mathbb{I}(\textbf{a})$, where $\mathbb{I}(\textbf{a})=1$ if $a_n=1$ for some $n\in \mathcal{N}$ (at least one replacement is performed) and $\mathbb{I}(\textbf{a})=0$ if $a_n=0$ for all $n\in \mathcal{N}$ (no replacements are performed). The one-step immediate cost is then

(3)\begin{align} c(\bf{x},\ell,\bf{a})= \mathbb{I}(\bf{a})c_s + \sum_{n\in \mathcal{N}}a_n\left[c_r +c_d(\ell)\right]+ \sum_{n\in \mathcal{N}}(1-a_n)\,c_d(\ell)\,\tau_n(x_n,\ell). \end{align}

The first term on the RHS of Equation (3) represents the setup cost. The second term is the sum of replacement costs and power production losses incurred for all replaced turbines. Finally, the third term accounts for the downtime costs estimated by the predictive functions for turbines that are not replaced.

All immediate costs are discounted at a fixed rate λ $(0 \lt \lambda \lt 1)$. The objective is to minimize the expected total discounted setup, replacement, and downtime costs over an infinite horizon. Let $\mathcal{P}$ denote the set of all nonanticipative replacement policies. Starting in state $(\textbf{x},\ell)$, the DM solves

(4)\begin{equation} V(\textbf{x},\ell)=\inf_{\pi\in \mathcal{P}}\left\{\mathbb{E}\left[\sum_{m=0}^{\infty}\lambda^m c(\textbf{x}_m,\ell_m,\textbf{a}_{\pi}(\textbf{x}_m,\ell_m)|(\textbf{x},\ell))\right]\right\},\qquad (\textbf{x},\ell)\in \Gamma, \end{equation}

where $\textbf{a}_{\pi}(\textbf{x}_m,\ell_m)$ denotes the action prescribed by policy $\pi\in \mathcal{P}$ in state $(\textbf{x}_m,\ell_m)$ at decision epoch m. Equation (4) is equivalent to

(5)\begin{equation} V(\textbf{x},\ell)= \min_{\textbf{a}\in\mathcal{A}}\left\{c(\textbf{x},\ell,\textbf{a})+\lambda \sum_{k\in \mathcal{L}}V(\hat{\textbf{x}}(\textbf{x},\ell,\textbf{a}),k)p_{\ell k}\right\},\qquad (\textbf{x},\ell)\in \Gamma, \end{equation}

where $\hat{\textbf{x}}(\textbf{x},\ell,\textbf{a})=\left({\hat{\textbf{x}}}_1(x_1,\ell,a_1),\ldots,{\hat{\textbf{x}}}_N(x_N,\ell,a_N)\right)$ is the N-dimensional vector of predicted degradation levels at the start of the next period. Note that the state space Γ is Borel-measurable, and the action space $\mathcal{A}$ is finite. Additionally, the immediate costs are strictly positive and bounded, and the problem is discounted. Therefore, by Theorems 6.2 and 6.3 of [54], there exists an optimal, deterministic stationary replacement policy, and the value iteration (VI) algorithm converges to the optimal value. Next, define the optimal action in state $(\textbf{x},\ell)$ by

(6)\begin{equation} \textbf{a}^*(\textbf{x},\ell):=\min\left\{\textbf{a}^{\prime}\in \mathrm{argmin}_{\textbf{a} \in \mathcal{A}}\left\{c(\textbf{x},\ell,\textbf{a})+\lambda \sum_{k\in \mathcal{L}}V(\hat{\textbf{x}}(\textbf{x},\ell,\textbf{a}),k)p_{\ell k}\right\}\right\}, \qquad (\textbf{x},\ell) \in \Gamma. \end{equation}

That is, if there exist multiple optimal actions in state $(\textbf{x},\ell)$, we take the minimum action; hence, Equation (6) has a unique solution. In Section 3, we establish structural properties and provide some practical insights into this important operational problem.

3. Structural results and insights

In this section, we establish some properties of the value function $V(\textbf{x},\ell)$ and the optimal replacement policy of the MDP model presented in Section 2. Before proceeding to the main results, we first provide two lemmas that describe useful properties of the immediate and downtime costs. The proofs of all results are contained in the Appendix.

Lemma 3.1. For each $n\in \mathcal{N}$, $\tau_n(x_n,\ell)$ is monotone nondecreasing in $x_n\in [0,1]$ and $\ell\in \mathcal{L}$.

Lemma 3.1 implies that, for each turbine n, the first crossing time to the failure threshold is shorter starting from higher degradation and environment states. Thus, the resulting downtime within a period is longer. Before proceeding to Lemma 3.3, we first review the notion of discrete concavity, which is more fully elucidated by Murota et al. [45, 46].

Definition 3.2. Let $X\subseteq \mathbb{Z}$. A function $f:X\to \mathbb{R}$ is said to be discrete concave on its domain X if and only if $X\ne \emptyset$ and

\begin{equation*}f(x-1)+f(x+1) \le 2f(x)\quad\text{for all}\ x\in X.\end{equation*}

Discrete concavity is useful for characterizing structural properties of the value function. Due to the nature of power output of wind turbines, henceforth, we assume that the downtime cost rate $c_d(\ell)$ is discrete concave on its domain $\mathcal{L}$. Furthermore, to simplify notation, let $\delta_n(x_n,\ell):=c_d(\ell)\cdot \tau_n(x_n,\ell)$ for each $n\in \mathcal{N}$. We first examine properties of the immediate cost function, $c(\textbf{x},\ell,\textbf{a})$.

Lemma 3.3. For each $\textbf{a}\in \mathcal{A}$, the immediate cost function $c(\textbf{x},\ell,\textbf{a})$ is

1. (a) monotone nondecreasing in $(\textbf{x},\ell)\in \Gamma$ and

2. (b) discrete concave on $\mathcal{L}$ for each $\textbf{x}\in \mathcal{X}$ if $\delta_n(x_n,\ell)$ is discrete concave on $\mathcal{L}$ for each $n\in \mathcal{N}$.

Lemma 3.3(a) asserts that the immediate costs are monotone increasing as the degradation level and environment state increase. That is, it is more costly to operate a system that is more degraded and/or operating in a more detrimental environment. However, Lemma 3.3(b) asserts that the marginal cost increases are diminishing in these two dimensions if $\delta_n(x_n,\ell)$ is discrete concave on its domain. While this assumption may appear restrictive on first glance, it is appropriate in this context due to the nature of power output as a function of wind speed and other relevant conditions. Next, we present structural properties of the value function.

3.1. Value function properties

We now examine important properties of the value function $V(\textbf{x},\ell)$. Proposition 3.5 posits that the monotonicity and discrete concavity of Lemma 3.3 extend to the value function if the transition matrix P possesses the increasing failure rate (IFR) property.

Definition 3.4. Let $\mathbf{P}=[p_{\ell k}]$ be the one-step TPM of a DTMC with state space $\mathcal{L}=\{1,\ldots,L\}$. Then P is said to be IFR if

\begin{equation*}\eta_k(\ell):=\sum_{\ell'=k}^{L}p_{\ell \ell^{\prime}}\end{equation*}

is nondecreasing in $\ell\in \mathcal{L}$ for each $k\in \mathcal{L}$.

Definition 3.4 implies that, starting from a higher environment state, the DTMC is more likely to transition to even higher states. Next, we state the main result related to $V(\textbf{x},\ell)$.

Proposition 3.5. If P is IFR, then $V(\textbf{x},\ell)$ is

1. (a) monotone nondecreasing in $(\textbf{x},\ell) \in \Gamma$ and

2. (b) discrete concave on $\mathcal{L}$ for each $\textbf{x}\in \mathcal{X}$ if $\delta_n(x_n,\ell)$ is discrete concave on $\mathcal{L}$ for each $n\in \mathcal{N}$.

Proposition 3.5(a) asserts that if P is IFR, then the cost starting from state $(\textbf{x},\ell)$ is nondecreasing over the entire state space. That is, for a fixed environment state $\ell$, the cost function is monotone nondecreasing in degradation x, and for a fixed level of degradation x, the cost is monotone nondecreasing in $\ell$. However, the costs do not increase without bound. Proposition 3.5(b) asserts that the marginal costs diminish as the environment state increases, a property that follows directly from the discrete concavity of the downtime cost rate function $c_d(\ell)$ on $\mathcal{L}$.

Naturally, one would like to examine the sensitivity of the value function to changes in the model’s cost parameters. For example, how sensitive is the value function to an increase in the setup cost c_s in light of the fact that this substantial cost can be shared by performing group replacements? To answer this question, we employ the notion of comparative statics, as described by Milgrom and Shannon [43]. Let w be any one of the nonnegative cost parameters c_s, c_r, or $c_d(\ell)$. The functions $c(\textbf{x},\ell,\textbf{a},w)$ and $V(\textbf{x},\ell,w)$ denote, respectively, the immediate cost and value functions associated with parameter w. Proposition 3.6 asserts monotonicity in all three cost parameters.

Proposition 3.6. If $c(\textbf{x},\ell,\textbf{a},w)$ is monotone nondecreasing in w for each $(\textbf{x},\ell)\in \Gamma$ and $\textbf{a}\in \mathcal{A}$, then $V(\textbf{x},\ell,w)$ is monotone nondecreasing in w.

Consider a case in which the DM can reduce the setup cost by decreasing the cost of preparing the JUV and helicopter. Proposition 3.6 asserts that optimally replacing the turbines incurs less cost in this case, that is, $V(\textbf{x},\ell,c_s)\le V(\textbf{x},\ell,c^{\prime}_s)$ for each $(\textbf{x},\ell)\in \Gamma$ and any $c_s\le c^{\prime}_s$. Next, we provide useful structural properties of the optimal wind farm replacement policy.

3.2. Characterizing the optimal replacement policy

Here, we present the main results of this section by establishing structural properties of the optimal replacement policy for a wind farm. For notational convenience, let

\begin{equation*}\mathcal{V}(\textbf{x},\ell):=\{n\in \mathcal{N}:a^*_n(\textbf{x},\ell) = 1\},\end{equation*}

be the set of all turbines for which replacement is optimal in state $(\textbf{x},\ell)$. Similarly, let $\mathcal{N}\setminus \mathcal{V}(\textbf{x},\ell)$ be the set of turbines for which it is optimal to do nothing in state $(\textbf{x},\ell)$.

Theorem 3.7 For each $(\textbf{x},\ell)\in \Gamma$, consider a degradation vector $\textbf{y}\in \mathcal{X}$ such that $y_n\ge x_n$ for $n\in\mathcal{V}(\textbf{x},\ell)$ and $y_n=x_n$ for $n\in\mathcal{N}\setminus \mathcal{V}(\textbf{x},\ell)$. If P is IFR, then $\textbf{a}^*(\textbf{y},\ell)=\textbf{a}^*(\textbf{x},\ell)$.

Theorem 3.7 asserts that, for a given environment state $\ell$, the replacement region extends in each dimension for which it is optimal to replace in degradation level x. The result is elucidated by way of a simple example. Consider a wind farm comprising three turbines (N = 3), and suppose that in state $(\textbf{x},\ell)=(0.5,0.5,0.5,\ell)$, it is optimal to replace turbines 1 and 2 but not to replace turbine 3, that is, $\textbf{a}^*(\textbf{x},\ell)=(1,1,0)$. Figure 4(a) illustrates that these same actions are optimal for any state $(\textbf{y},\ell)$ in the shaded region (extension of the optimal actions in two dimensions).

Figure 4. Extension of optimal actions at degradation level x to higher degradation levels. (a) $\textbf{a}^*(\textbf{x},\ell)=(1,1,0)$ in shaded region. (b) $\textbf{a}^*(\textbf{x},\ell)=(1,1,1)$ in shaded region.

That is, $\textbf{a}^*(\textbf{y},\ell)=(1,1,0)$ for any degradation level $\textbf{y}\in [0.5,1]\times[0.5,1]\times \{0.5\}$. Figure 4(b) illustrates a different scenario in which the state is again $(\textbf{x},\ell)=(0.5,0.5,0.5,\ell)$, but it is optimal to replace all three turbines, that is, $\textbf{a}^*(\textbf{x},\ell)=(1,1,1)$. For this case, we see that action $(1,1,1)$ is optimal for any degradation level $\textbf{y}\in [0.5,1]\times[0.5,1]\times [0.5,1]$, so the optimal actions extend in three dimensions to states with higher levels of degradation.

Now for a given environment state $\ell$, consider a single turbine $n^{\prime}\in \mathcal{N}$ with degradation level $x_{n^{\prime}}$ and fix the degradation levels of all other turbines $n\in \mathcal{N}\setminus \{n^{\prime}\}$. The following corollary characterizes the behavior of the optimal action $a^*_{n^{\prime}}(\textbf{x},\ell)$ for each environment state.

Corollary 3.8. Suppose that P is IFR and let $n^{\prime}\in \mathcal{N}$. Fix $\ell \in \mathcal{L}$ and $x_n\in [0,1]$ for each $n\in \mathcal{N}\setminus \{n^{\prime}\}$. Then the optimal action $a^*_{n^{\prime}}(\textbf{x},\ell)$ is monotone nondecreasing in $x_{n^{\prime}}$.

As a consequence of economic dependence, it is possible that replacement of turbine n ^ʹ is not optimal for any level of degradation, that is, a degradation-based threshold policy does not exist. However, if reactive replacement is required whenever unit n ^ʹ is found to be failed, Corollary 3.8 implies the existence of a degradation threshold $x^*_{n^{\prime}}$ such that

(7)\begin{equation} a^*_{n^{\prime}}(\bf{x},\ell)= \begin{cases} 1, & \text{if}\, x_{n^{\prime}}\ge x^*_{n^{\prime}},\\ 0, & \text{if}\, x_{n^{\prime}} \lt x^*_{n^{\prime}}. \end{cases} \end{equation}

Equation (7) mirrors the optimal replacement policy described in Theorem 2 of [71] for a single degrading system that is influenced by its random environment; however, their model assumes that an immediate, reactive replacement is performed whenever a failure is revealed. Our model shows that, for some turbines, delaying replacement may be optimal due to the economic dependence induced by the shared setup cost (c_s) and the turbines’ stochastic dependence induced by their common environment. That is, the threshold for replacing a turbine may depend not only that turbine’s degradation but also on the degradation levels of the other turbines in the wind farm. This interesting feature of our model is more transparent in the case of N = 2 turbines.

3.3. Special case: N = 2

Structural properties of the optimal replacement policy over the entire space Γ are not immediately obvious when a wind farm contains three or more turbines. Fortunately, Theorem 3.7 can be used to establish the dependence of an optimal replacement threshold of a particular turbine on the degradation of the other turbine in a two-turbine wind farm (N = 2). Before formalizing this interesting result, it is helpful to consider a graphical depiction of the two-turbine case in Figure 5. For $\ell$ fixed, the black circles in Figure 5 represent degradation levels $(x_1,x_2)\in [0,1]\times [0,1]$. From Theorem 3.7, we can deduce the following. If the optimal action at point A is $(0,1)$, then $(0,1)$ remains optimal for any point along the line segment AD. Specifically, for a fixed degradation level x ₁, if it is optimal to replace turbine 2 at A, then it is optimal to replace turbine 2 for each point along AD. Similarly, if the optimal action at point B is $(1,0)$ (replace turbine 1 but not turbine 2), then $(1,0)$ remains optimal for any point along the line segment BE. Finally, at point C, the optimal action is $(1,1)$, replace both turbines. Then it is also optimal to replace both turbines for every point contained in the shaded region.

Figure 5. Extension of the optimal actions for two turbines in a fixed environment state.

Next, we seek to provide structure to replacement regions over the degradation space $\mathcal{X}$. For a fixed environment state $\ell\in \mathcal{L}$, let $x^*_2(x_1,\ell)$ denote the optimal replacement threshold for turbine 2 when the degradation of turbine 1 is $x_1\in [0,1]$. Similarly, let $x^*_1(x_2,\ell)$ be the optimal replacement threshold for turbine 1 when the degradation of turbine 2 is $x_2\in [0,1]$. When they exist, let

(8)\begin{align} b_1(x_2,\ell)&:=\inf\{x_1\ge 0:\textbf{a}^*(x_1,x_2,\ell)=(1,1)\}, \end{align}

(9)\begin{align} b_2(x_1,\ell)&:=\inf\{x_2\ge 0:\textbf{a}^*(x_1,x_2,\ell)=(1,1)\}. \end{align}

That is, $b_1(x_2,\ell)$ ($b_2(x_1,\ell)$) is the minimum level of turbine 1 (turbine 2) degradation that makes group replacement optimal when the degradation of turbine 2 (turbine 1) is x ₂ (x ₁). The next proposition asserts that the optimal replacement threshold of one turbine is monotone decreasing in the degradation level of the other turbine beyond these special points $b_1(1,\ell)$ and $b_2(1,\ell)$.

Proposition 3.9. For each $\ell\in \mathcal{L}$, if P is IFR and $\textbf{a}^*(1,1,\ell)=(1,1)$, then

1. (a) $x^*_2(x_1,\ell)$ is monotone nonincreasing in $x_1\in \{x_1: b_1(1,\ell)\le x_1\le 1\}$ and

2. (b) $x^*_1(x_2,\ell)$ is monotone nonincreasing in $x_2\in \{x_2: b_2(1,\ell)\le x_2\le 1\}$.

Proposition 3.9 establishes the fact that the replacement threshold of one turbine depends on the degradation state of the other turbine. Figure 6 depicts the region in which it is optimal to replace turbine 2, that is, $\left\{(x_1,x_2):\textbf{a}^*(x_1,x_2,\ell)\in \{(0,1),(1,1)\}\right\}$. The figure illustrates that the replacement threshold for turbine 2 is monotone decreasing in x ₁ for any $x_1\ge b_1(1,\ell)$.

Figure 6. Monotone threshold policy for turbine 2 in a wind farm with two turbines.

Figure 6 also highlights the fact that it is advisable for the DM to decrease the optimal threshold of turbine 2 as turbine 1 reaches higher degradation levels. The results in Proposition 3.9 confirm that the replacement thresholds for both turbines are monotone decreasing. In Section 4, we provide some numerical examples illustrating the effects of economic and stochastic dependence on the optimal turbine replacement policies.

4. Numerical illustrations

Here, we illustrate the structure of optimal replacement policies for a two-turbine wind farm using our MDP model. While our main results are valid for any finite number of turbines, we choose to illustrate the policies for N = 2 since they can be easily visualized in the xy-plane. It is worth noting that, although the state space grows exponentially in the number of turbines N, approximate dynamic programming (ADP) and state-space reduction techniques can be employed to solve larger instances, as was done in [2]. First, we provide numerical examples demonstrating how these policies differ depending on the magnitude of the setup cost (c_s) and the prevailing environment state ($\ell$). Second, we assess the importance of incorporating economic and stochastic dependence when evaluating turbine replacement policies. To that end, we consider simpler models that either ignore these features altogether or use fewer environment states than are needed to adequately characterize the environment. We begin by describing a baseline example, which considers a wind farm with two wind turbines. There exists ample evidence to suggest that 6–8 states are sufficient to model environmental conditions [4]; hence, our baseline example contains six distinct environment states.

4.1. A baseline example

Suppose there are two turbines in an offshore wind farm (N = 2), and the turbines are exposed to an environment whose evolution is modeled by a DTMC with state space $\mathcal{L}=\{1,\ldots,6\}$ and TPM

\begin{equation*}\mathbf{P}= \begin{bmatrix} 0.40 & 0.20 & 0.20 & 0.14 & 0.05 & 0.01\\ 0.22 & 0.30 & 0.20 & 0.15 & 0.10 & 0.03\\ 0.20 & 0.14 & 0.30 & 0.22 & 0.11 & 0.03\\ 0.10 & 0.10 & 0.20 & 0.35 & 0.15 & 0.10\\ 0.05 & 0.08 & 0.10 & 0.20 & 0.35 & 0.22\\ 0.03 & 0.05 & 0.10 & 0.15 & 0.40 & 0.27\\ \end{bmatrix}.\end{equation*}

It can be shown that P possesses the IFR property. For this illustration, it is assumed that, for each turbine $n\in \mathcal{N}$, the DM predicts future degradation using the linear predictive function

\begin{equation*}f_n(x_n,\ell,t) = x_n + r_n(\ell)t,\quad t\ge 0,\end{equation*}

where $r_n(\ell)$ is the degradation rate of turbine n in environment state $\ell\in \mathcal{L}$. The form of f_n is immaterial as long as it satisfies the nonnegativity and monotonicity assumptions stated in Section 2. It is difficult to obtain proprietary wind farm O&M records to parameterize our model. Therefore, notional setup costs, downtime cost rates, and degradation rates were chosen relative in scale but are not intended to reflect any real data provided to the authors. The linear degradation rates were chosen such that $r_n(i) \lt r_n(j)$ whenever i < j, and their magnitudes are in proportion to the failure threshold ξ = 1. Because the time scale of one period is on the order of 1 week, the rates of degradation are relatively small as compared to the failure threshold of 1. Similarly, the downtime cost rates are selected such that $c_d(i) \lt c_d(j)$ whenever i < j. The setup and replacement costs are $c_s=5$ and $c_r=8$, respectively. Table 1 lists all other notional parameter values.

Table 1. Downtime cost and degradation rates for each environment state.

To implement the VI algorithm of [54], we discretized the continuous degradation space $\mathcal{X}$ (cf. Kushner and Dupuis [37]). Specifically, for each $n\in \mathcal{N}=\{1,2\}$, the degradation interval $[0,1]$ was uniformly discretized into 101 states so that $x_n\in \{0.00,0.01,\ldots,0.99,1.00\}$, and we let $\mathcal{X}^{\prime}:=\{0.00,0.01,\ldots,0.99,1.00\}^2$. Then for each $(\textbf{x},\ell)$ in $\mathcal{X}^{\prime}\times \mathcal{L}$, let $v^k(\textbf{x},\ell)$ denote the kth iterate of the VI algorithm with $v^0(\textbf{x},\ell)=0$. The algorithm terminates at iteration k $(k\in \mathbb{N}$) if

\begin{equation*}\big\|v^{k} -v^{k-1}\big\|_{\infty} = \max_{(\textbf{x},\ell)}\left\{\big|v^{k}(\textbf{x},\ell)-v^{k-1}(\textbf{x},\ell)\big|\right\}\le \epsilon_v,\end{equation*}

where $\epsilon_v \gt 0$ is a tolerance level. For each implementation of the VI algorithm, we set $\epsilon_v = 0.01$ and λ = 0.99. The VI algorithm was coded in Python 3.9 and executed on a personal computer with a 3.40 GHz processor and 8 GB of RAM. Next, we examine the sensitivity of the optimal replacement policy to each environment state and the setup cost.

4.2. Baseline example sensitivity

Starting in state $(x_1,x_2,\ell)=(0,0,1)$, the VI algorithm yields an optimal total discounted cost of 3,044.20 for the baseline example. For a fixed environment state, replacement regions are the regions in which at least one turbine is replaced (shaded regions). The opportunistic region is the region in which both turbines are replaced (dark shaded region). Figure 7 depicts the replacement regions in different environment states.

Figure 7. Baseline example optimal replacement policy for each environment state ($c_s=5$): (a) environment state $\ell=1$, (b) environment state $\ell=2$, (c) environment state $\ell=3$, (d) environment state $\ell=4$, (e) environment state $\ell=5$, and (f) environment state $\ell=6$.

Figure 7(b)–(f) shows that higher environment states lead to larger replacement regions, as these states imply accelerated degradation and higher power production rates. Hence, in higher environment states, the DM opts to lower the replacement threshold to avoid high production losses caused by turbine failures. However, in Figure 7(a), the largest replacement region is for environment state $\ell=1$ since the production and degradation rates for this state are least among all environment states; hence, the DM prefers to perform the preventive replacements much sooner. Such a scenario may correspond to the overhaul of large, critical components in a wind farm, as major replacements are typically performed in the summer months when environment conditions are milder. Figure 7(a) also illustrates the result in Proposition 3.9—the optimal replacement threshold of one turbine is decreasing in the degradation state of the other turbine when both are in the opportunistic region. However, we also observe nonintuitive behavior for the optimal replacement policy in Figure 7; the optimal replacement threshold of each turbine is not necessarily decreasing over the entire degradation state space. For example, in Figure 7(a), for a fixed degradation state of turbine 2 ($x_2=0.4$), the DM opts to replace turbine 2 at point B but not to replace at points A and C. Thus, by increasing the degradation state of turbine 1, the optimal replacement threshold for turbine 2 decreases first (moving from point A to B) and then increases (moving from point B to C). One possible explanation for this behavior is that, at point C, turbine 2 is closer to the opportunistic region and will reach this region sooner. Thus, it is best for the DM to postpone replacement of turbine 2 until turbine 1 further degrades so that a group replacement can be performed and setup costs can be shared.

In the first instance of the baseline example, the setup cost ($c_s=5$) was relatively low compared to the downtime cost rates. This scenario corresponds to the case when the required resources for replacement are available (e.g., no cost for chartering a JUV and helicopter) and the majority of the setup cost is fueling. For the sake of comparison, we also examined a scenario with a relatively high setup cost corresponding to the case when a helicopter and JUV must be chartered from a spot market, and the total setup cost exceeds the highest capacity of power production for a turbine ($c_s = 80$). Figure 8 shows how the optimal replacement regions shrink with a substantial increase in the setup cost. Figure 8(c)–(f) shows that for some degradation levels, the DM may allow the other turbine to remain failed and postpone replacement. That is, for higher states of degradation, replacing only one turbine is not optimal, and the DM is more inclined to perform group replacement due to the magnitude of the setup cost.

Figure 8. Baseline example optimal replacement policy for each environment state ($c_s=80$): (a) environment state $\ell=1$, (b) environment state $\ell=2$, (c) environment state $\ell=3$, (d) environment state $\ell=4$, (e) environment state $\ell=5$, and (f) environment state $\ell=6$.

One would expect the opportunistic region to be larger in higher environment states since the DM opts to perform the group replacement sooner to avoid higher power production losses. However, in Figures 7 and 8, the opportunistic region does not necessarily grow with the environment state. There are two reasons for this nonintuitive behavior. First, if the environment state is 1, then the likelihood of transitioning to higher states is low. Hence, it is expected that, in subsequent periods, the environment remains in state 1 with a higher probability. Second, the downtime cost in environment state 1 is low; hence, the DM prefers to replace both turbines in an environment state that leads to a lower production loss. In the next subsection, we show how models with incorrect assumptions impact the optimal replacement policies.

4.3. Value of correct modeling

Here, our aim is to assess the magnitude of cost increases resulting from incorrect model assumptions. The MDP model incorporates economic dependence via the setup cost c_s and stochastic dependence via the association of degradation processes to the turbines’ common environment. Here, we compare the expected total discounted costs obtained by two distinct models and compare those with the costs obtained by the model of Section 2. In the first model, the DM simply ignores both types of dependence and assumes that replacement decisions are made for each turbine independently. In the second model, the DM includes dependence but assumes the environment can be described by a simple two-state DTMC, thereby ignoring any intermediate environment states.

Let $\pi^*$ be the optimal replacement policy obtained by the MDP model of Section 2, and let π_i be the optimal replacement policy obtained by Model i, for $i=1,2$. Table 2 summarizes how the models differ with regard to their underlying assumptions.

Table 2. Summary of the three distinct models for the replacement problem.

The full model incorporates both economic and stochastic dependence and all six environment states. After obtaining the optimal policy for each model using the VI algorithm, we simulated the respective costs of the modified policies (π ₁ and π ₂) on the full model using a large number of sample paths. Along each sample path, the total discounted cost was computed, and these values were compared. The simulation run length, which corresponds to the number of decision epochs, is denoted by M ($M\in \mathbb{N})$. Since the expected one-step costs are bounded, and the cost function is discounted, the simulation run length can be determined a priori to ensure that the total discounted cost is accurate to a fixed constant. The simulation run length M was chosen such that $M\ge \left[\ln((1-\lambda)\epsilon_s/C)/\ln(\lambda)\right]-1$, where C is any valid upper bound on the expected one-step costs and λ is the discount factor. For all numerical examples, M was chosen to correspond to $\epsilon_s = 0.01$, λ = 0.99, and $C=c_s+N[c_r+c_d(L)]$. For the baseline example, $M\ge 1428$, so we chose M = 1500.

To compute the average simulated total discounted cost under each policy, we selected the number of replications (R) to achieve a desired margin of error using the well-known relationship

(10)\begin{equation} R = \left(z_{\alpha/2}\,\hat{\sigma}/\Delta\right)^2, \end{equation}

where $z_{\alpha/2}$ is the critical z-value above which lies area $\alpha/2$ under the standard normal density function, $\hat{\sigma}$ is the estimated standard deviation, and Δ is the desired margin of error. For each of the following models, we chose a 95% confidence level (α = 0.05, $z_{\alpha/2}=1.96$) and $\Delta=1$. The simulation models were coded in the MATLAB R2022a computing environment and executed on a personal computer with an 8-core CPU and 8 GB of RAM. We next describe the modified MDP formulations for Models 1 and 2, along with the corresponding numerical results.

Model 1. The first modified model ignores both economic and stochastic dependence; therefore, the problem can be decomposed into two independent, single-turbine problems. For each decomposed problem, the setup and replacement costs are assessed every time a replacement is performed. Although both turbines are assumed to operate in environments governed by the same transition matrix P, their environments (and degradation processes) evolve independently. Thus, for turbine $n\in \{1,2\}$, the state is $(x_n,\ell_n)\in [0,1]\times \mathcal{L}$, $a_n\in \{0,1\}$ is the action, and the immediate cost function is

\begin{equation*}c(x_n,\ell_n,a_n) = a_n\left[c_s +c_r +c_d(\ell_n)\right]+(1-a_n)c_d(\ell_n)\tau_n(x_n,\ell_n).\end{equation*}

The Bellman equation for the decomposed MDP model n is

\begin{equation*}V(x_n,\ell_n)=\min_{a_n\in \{0,1\}}\left\{c(x_n,\ell_n,a_n)+\lambda \sum_{k_n\in \mathcal{L}}V({\hat{\textbf{x}}}_n(\ell_n,a_n),k_n)p_{\ell_n k_n}\right\},\qquad (x_n,\ell_n)\in [0,1]\times\mathcal{L},\end{equation*}

where

\begin{equation*}{\hat{\textbf{x}}}_n(\ell_n,a_n)= \begin{cases} \min\{1,f_n(x_n,\ell_n,1)\}, & \text{if}\ a_n=0,\\ 0, & \text{if}\ a_n=1. \end{cases}\end{equation*}

Figure 9 illustrates the behavior of the Model 1 optimal replacement policy (π ₁) for each environment state. We observe that the optimal replacement threshold of one turbine does not depend on the other’s degradation state in the absence of economic and stochastic dependence. Moreover, the optimal replacement thresholds decrease with the environment state. That is, in environments that induce higher degradation and power production rates, preventive replacements are prescribed at lower levels of degradation for both turbines.

Next, we seek to assess how the Model 1 optimal policy (π ₁) performs under the assumptions of the full model (i.e., when economic and stochastic dependence are present). To this end, we apply policy π ₁ to a simulation of the full model and compare its resulting simulated total discounted cost. We first performed 5,000 replications of the simulation model (each with $M=1,500$ decision epochs) and obtained a sample standard deviation $\hat{\sigma}\approx256$. By Equation (10), the number of replications needed to ensure $\Delta=1$ is $R=251,763$; hence, we selected $R=252,000$. The policy π ₁ was applied in the simulation of the full model, and the simulated total discounted costs for three different initial states are reported in Table 4. The simulated costs are also compared with those obtained using VI on the full model, which includes both economic and stochastic dependence.

Figure 9. Baseline example optimal replacement policy for each environment state ($c_s=80$): (a) environment state $\ell=1$, (b) environment state $\ell=2$, (c) environment state $\ell=3$, (d) environment state $\ell=4$, (e) environment state $\ell=5$, and (f) environment state $\ell=6$.

Model 2. Like the full model, this second modification incorporates both economic and stochastic dependence (via shared setup costs and a common environment). However, in this case, the DM lacks sufficient information to assess the number of environment states and the degradation rates associated with each state. Recall that the full model assumes that the environment can transition among six distinct states. In Model 2, the DM uses only two states to describe the environment ($\mathcal{L} = \{1,2\}$) and estimates a degradation rate for each state. Table 3 summarizes the downtime cost rates and degradation rates corresponding to each environment state. Note that the values used for states 1 and 2 in Model 2 correspond to those of the extreme states 1 and 6 from Table 1. For this new two-state model, the DM scales the original transition probabilities, p ₁₁, p ₁₆, p ₆₁, and p ₆₆, to construct the new TPM

\begin{equation*}\widehat{\mathbf{P}}=\begin{bmatrix} 0.98 & 0.02\\ 0.10 & 0.90\\ \end{bmatrix},\end{equation*}

Table 3. Downtime cost and degradation rates for the environment states of Model 2.

where, for example, $\hat{p}_{11}$ was obtained by $\hat{p}_{11}=p_{11}/(p_{11}+p_{16})\approx 0.98$. All other components of this MDP model, including the immediate cost function and the Bellman equation, are identical to the MDP model defined in Section 2. Figure 10 depicts the optimal replacement policy (π ₂) obtained by VI algorithm for each of the two environment states. To compare the costs of this policy to the full model, it was necessary to create a mapping between each of the six states in the full model to the two states of the modified environment process. Specifically, the replacement policy for $\ell=1$ (Figure 10(a)) was used for states 1, 2, and 3 in the full model. The replacement policy for $\ell=2$ (Figure 10(b)) was used for states 4, 5, and 6 in the full model.

Figure 10. Model 2 optimal replacement policy for each environment state. (a) Environment state $\ell=1$. (b) Environment state $\ell=2$.

We first performed 5,000 replications of the simulation model (each with $M=1,500$ decision epochs) and obtained a sample standard deviation $\hat{\sigma}\approx 251$. By Equation (10), the number of replications needed to ensure $\Delta=1$ is $R=242,024$; hence, we selected $R=243,000$. As before, we apply policy π ₂ to a simulation of the full model and compare its resulting simulated total discounted cost. The simulated total discounted costs for six different initial states are summarized in Table 4 and compared with those obtained using VI on the full model.

Table 4 compares the expected total discounted costs of the full model and the simulated total discounted costs of Models 1 and 2 starting from initial states $(0,0,1)$, $(0,0,2)$, $(0,0,3)$, $(0,0,4)$, $(0,0,5)$, and $(0,0,6)$. This table reveals several interesting results. First, the policies resulting from Model 1, in which both economic and stochastic dependence are ignored, yield average total discounted costs that are just over 8% higher than those generated by the full model, indicating the importance of sharing setup costs and consideration of how the turbines’ degradation processes are related via their shared environment. Second, we note that the policies resulting from Model 2, which includes both types of dependence but fails to adequately characterize the environment process, result in cost increases on the order of 5%. These results provide evidence that accounting for all possible environment scenarios can be important when assessing the lifetime replacement costs of an offshore wind turbine. Finally, we observe a general trend that the total costs are increasing in the initial environment state (assuming both turbines start in an as-good-as-new condition). This observation is in agreement with Proposition 3.5(a).

Table 4. Total discounted costs for six distinct initial states.

4.4. Comparisons with heuristic policies

Here, we assess the performance of the optimal replacement policy obtained from our model with three distinct and common heuristic policies:

• • Reactive replacement policy. Using this policy, the DM replaces the component reactively, that is, immediately following a failure.

• • Age-replacement policy. The classical age-replacement policy prescribes replacement either at failure or at a chosen deterministic age T, whichever occurs first. The replacement age is taken to be the expected lifetime of the unit, which is typically the manufacturer’s mean time-to-failure.

• • Fixed interval policy. This policy performs a replacement every T units, whether a replacement is warranted or not. Here, the deterministic inter-replacement time T is taken to be the manufacturer’s mean time-to-failure.

We compared the total costs generated by the MDP policies with those of the heuristic policies starting from six distinct initial states (corresponding to the six environment states). The results in Table 5 show that the total discounted costs obtained by the MDP model are approximately 17% lower than those obtained using reactive replacement, 14% lower than those obtained using the age-replacement policy, and over 40% lower than those obtained using a fixed interval policy. These results highlight the value of using the optimal policy as compared to simple heuristic policies.

Table 5. Total discounted cost comparison: Optimal policy versus three heuristic policies.

5. Conclusions and discussion

We devised and analyzed an MDP model to optimally prescribe the replacement times of critical components in an offshore wind farm. Our model includes both economic and stochastic dependence between the turbines operating in the farm. Economic dependence is induced by significant setup costs that can be shared by performing opportunistic replacements; stochastic dependence exists in the turbines’ degradation processes due to their common exposure to a randomly varying environment. The main results may have significant practical implications for offshore wind farm operators who seek to reduce turbine downtime and ensure sustained power production capacity. We established the monotonicity of actions for each turbine in the farm (fixing the degradation of all other turbines). Most interesting is the fact that the optimal replacement time of one turbine depends not only on its own level of degradation but also on the degradation levels of the other turbines in the farm. Furthermore, beyond a certain point, the replacement thresholds are monotone. We showed that it may be more beneficial to delay replacements and reduce overall costs by performing group replacements when the setup costs can be shared among multiple turbines. This result is important since the cost of chartering or deploying service operations vessels and/or JUVs is significant. Numerical examples illustrated the structure of replacement policies in the case of two turbines and showed that monotone policies emerge beyond certain degradation thresholds for both turbines. We also demonstrated the value of including dependencies and accounting for the influence of the environment. When compared with three different heuristic policies, it was shown that the optimal policy substantially reduces the total discounted costs.

While the insights gained from the MDP model are valuable, it can be further expanded and improved to incorporate additional contextual information (in the form of constraints) related to the maintenance of offshore wind farms. Several factors influence the feasibility of performing maintenance operations, including the wind speed and wave height, availability of specialized resources (vessels and/or helicopters), and availability of crews to perform the work. Specifically, safety limits for wind speed and wave height dictate conditions for which vessels can safely operate in the sea, and when conditions are unsafe, operations are deemed infeasible. To incorporate this realism into our model, one need only to append an additional state—an indicator variable that assumes the value 0 or 1—indicating if weather conditions are suitable for maintenance operations. When conditions are too harsh, the indicator is assigned a value 0 and maintenance cannot be performed. In this case, the only feasible action is “do nothing” and wait until the next decision epoch. Another critical factor affecting the feasibility of maintenance is the availability of specialized resources (e.g., SOVs, CTVs, JUVs, and helicopters) or work crews. Likewise, these constraints can be included in the MDP model by adding indicator variables to the state vector, which indicate whether a resource or crew are available at each decision epoch. However, the inclusion of these additional components in the state vector increases the dimensionality of the problem to the extent that it may become intractable or the structural results might be lost. Another complication is that the number of states grows exponentially in the number of turbines in the wind farm. To deal with these issues, one can consider ADP techniques, along with state–space reduction strategies, as demonstrated by Abdul-Malak and Kharoufeh [2].

Although the model presented herein focused on the replacement of major components in an offshore wind turbine, it can be modified to prescribe the timing of other maintenance actions (e.g., minor or medium repairs) for components that experience more frequent failures than major components such as generators and gearboxes. The MDP model employs an assumption that replacement takes one period (up to 1 week) to complete and that all replacements are completed before the next decision epoch. Because the average time needed to repair or replace a minor component is typically shorter than that of a major component, the length of time between decision epochs can be chosen to ensure that the maintenance action is completed within one period. For example, minor repairs or replacements can usually be completed within 1 day, whereas major replacements may require 2–7 days to complete. Thus, the time scale of the MDP model can be modified to account for minor repairs or replacements by setting the period length to 1 day. It is important to note that the per-period costs and the degradation function must be modified in this case. It is expected that the opportunistic region for minor or medium repairs will be larger than the opportunistic region for replacing major components.

Finally, in our framework, the DM’s objective is to minimize the expected total discounted costs, which include the setup and replacement costs, as well as the costs associated with turbine downtime. However, if the objective is to maximize expected profit over a planning horizon, it is necessary to modify the MDP model by incorporating pricing and production rate information into the one-step profit. To this end, let $C_n(\ell)\in \mathbb{R_+}$ be the nominal rate of power production for turbine n when the environment state is $\ell$ and ρ_m ($\rho_m \gt 0$) the price at decision epoch m. It can be assumed that a turbine produces power at its nominal rate when it is operating. The price can be incorporated by expanding the state to $(\textbf{x},\ell,\rho)$ and using pricing information, for example, from regional transmission operators such as PJM (https://www.pjm.com) to estimate the probability that the price transitions from ρ_m to $\rho_{m+1}$ in one period. By including the dynamic and stochastic price, the DM may consider maximization of expected profits. During each period, revenue is only accrued for a turbine if it is operational; no revenue is included during downtime. For instance, at the beginning of a period, if turbine n is failed $(x_n=1)$ or is scheduled for replacement $(a_n=1)$, there is no earned revenue because the turbine is down for the entire period. However, if turbine n is operational at the start of period m and no replacement is scheduled, the one-step revenue can be calculated as $C_n(\ell)\rho_m\left[1-\tau_n(x_n,\ell)\right]$, where $\tau_n(x_n,\ell)$ is the predicted downtime during that period. Finally, the profit in each period is determined by subtracting the applicable setup and replacement costs (if there is a need for replacement) from the generated revenues.