Robotic autonomous systems for earthmoving equipment operating in volatile conditions and teaming capacity: a survey

Abstract

There has been an increasing interest in the application of robotic autonomous systems (RASs) for construction and mining, particularly the use of RAS technologies to respond to the emergent issues for earthmoving equipment operating in volatile environments and for the need of multiplatform cooperation. Researchers and practitioners are in need of techniques and developments to deal with these challenges. To address this topic for earthmoving automation, this paper presents a comprehensive survey of significant contributions and recent advances, as reported in the literature, databases of professional societies, and technical documentation from the Original Equipment Manufacturers (OEM). In dealing with volatile environments, advances in sensing, communication and software, data analytics, as well as self-driving technologies can be made to work reliably and have drastically increased safety. It is envisaged that an automated earthmoving site within this decade will manifest the collaboration of bulldozers, graders, and excavators to undertake ground-based tasks without operators behind the cabin controls; in some cases, the machines will be without cabins. It is worth for relevant small- and medium-sized enterprises developing their products to meet the market demands in this area. The study also discusses on future directions for research and development to provide green solutions to earthmoving.

1. Introduction

The application of automation in construction and mining has brought significant contributions with advances of robotic autonomous systems (RASs) usually on a confined and predefined working site, taking advantage of preprogrammed mobility and well-known interaction with the static environment. The highly complex and variant nature of construction and mining tends to reduce reliable performance of pre-optimized machines which are sensitive with the working environment [1]. Moreover, harsh and difficult conditions as well as changes in the working environment also require RAS-based systems to yield earthmoving process the ability to effectively respond to any disruptions with a continuous demand in productivity. In this regard, there has been increasing interest in research and development for RAS technologies in dealing with volatile environments and enhancing the teaming capability of earthmoving equipment. The implemented RAS technologies cover such areas as perception, localization, navigation, control, communication, and interactive cooperation.

Earthmoving equipment, including excavators, bulldozers, front-end loaders, and graders, is discussed in this paper. Since the automated systems for these machines have functions in common, in this paper, they are referred to as platforms. Autonomous operations of those construction machines, for earthworks often including a mobile platform and an attachment performed like in unmanned ground vehicles (UGVs), can be classified in the five modes of control [2]. While RAS technologies have been applied to those platforms to increase work performance, productivity, safety, efficiency, and environmental sustainability [3], fully autonomous functions of earthmoving equipment are still limited and often require coordinative commands from human operators in unstructured work sites or from impacts of unexpected environment changes as well as extreme events.

In earthmoving, the typical machines like excavators and front-end loaders usually performs the three tasks during a single cycle of operation, namely (i) Loading, (ii) Navigating, and (iii) Dumping. Since this cycle in many applications is quite often repeated, task efficiency and safety of humans and machinery are of crucial importance. In dealing with weather extreme events and the need to improve task effciciency and work capacity, integrated RAS technologies can help with the platform traction and mobility, including path planning, collision avoidance, and navigation, and for the work attachment, effective and efficient control of the arm, boom and bucket for digging, loading, and dumping as well as various tasks such as land clearance, ditching, dirt bunding, compacting, road mending, and paving. In terms of research and developments, significant effort has however been continuously devoted to the important areas of: (i) modeling and control, including kinematics, dynamics, tool–soil interactions, actuator low-level control, compliance control, feedforward control, and intelligent control; (ii) laboratorial testings and full-scale experiments; (iii) machine and environment sensing, and robotic vision; (iii) localization and navigation; (iv) path planning and traction control; (iv) task decomposition and system architecture, and high-level control; and (v) communication and cooperative control.

The onsite environment for earthworks is directly subject to changes in weather conditions and climate, which affect the earthmoving process in general. In this article, undesired factors impacting equipment operation are known as volatile environments, where interactions with robots are difficult to model due to reasons such as dynamics, uncertainty, complexity, and unpredictability [4]. Indeed, the impact of adverse weather is a common cause of delays, legal claims, and economic losses in construction projects. Dealing with volatile environments, research and development should focus on considering weather effects in project planning for onsite activities, mapping predicted weather variables with construction productivity, and developing RAS-based measures to enhance resilience against abrupt, unpredictable weather changes in earthmoving activities or projects. In this regard, a technique for evaluating the difficulty level of the operating conditions of the equipment involved will be useful for smart management of its engine power [5].

Among the emerging trends is the development of systems for the management and tracking of a group of platforms, as well as machine guidance and control. Addressing the increasing requirements of automation, innovative research, and RAS-based implementation of earthmoving equipment in industry and academia are being conducted under such sub-categories as fleet management and facilities tracking, safety, machine control, pose estimation and tele-operation, as well as remote control and autonomous operations [6]. With the rapid growth of the telecommunication industry, the use of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication as well as advances in the Internet of Things (IoT) and cyberphysical systems (CPSs) will benefit earthmoving processes in various ways.

For earthmoving, a new paradigm has emerged on exploiting the pervasive presence of various machines possessing digital intelligence in a jobsite. RAS-based equipment can make themselves recognizable and can behave “intelligently” by making contextualized decisions through information aggregation and sharing with other machines for learning and improving automatically as the artificial intelligence (AI) algorithms. To this end, a multiple-robot system can accomplish certain earthmoving tasks that a single machine barely can and deliver them at a higher productivity, thus improving work capacity and task efficiency for earthmoving. Cooperative autonomous operations require the cooordination between equipment members to have the ability to: (i) interact with other machines performing the same or different tasks; (ii) perform a shared task in association with other machines; (iii) autonomously divide a task between several machines; and (iv) effectively manage and prioritize events.

Although RAS technologies have contributed to significant improvements of task performance and productivity of earthworks, the cooperative control of multiple platforms under adverse conditions of the environment has appeared to be a challenging area for further research and development. For example, heavy rainfall can make the onsite roads become slippery, causing collision of autonomous platforms [7]. The objectives of this survey is to provide a comprehensive review on the main RAS technologies applied to earthmoving machines in dealing with volatile environments and with the interactive coordination of multiple platforms operating onsite, with the use of robotic autonomous equipment, the state of the art of academic discussions as well as recent developments from industrial perspectives, and future directions on these topics. Apart from digesting scientific publications appearing in the last 5 years in academic peer-reviewed journals, magazines, books, and important websites, this paper also reflects an application-oriented focus for interested practitioners and stakeholders in the area.

This paper is organized as follows. After the introduction, Section 2 presents the challenges identified in the earthmoving processes as discussed in this paper and the modes of control for the platforms using the core RAS technologies for earthwork automation. Section 3 is devoted to the use of RAS technologies in dealing with volatile environments and Section 4 is focused on their application in cooperative tasks. Recent achievements in addressing these topics from industrial perpectives are presented in Section 5 along with discussion on future directions for research and development. Finally, Section 6 concludes the paper.

2. Earthmoving challenges and RAS-based modes of control

This survey is focused on two identified challenging topics in earthworks, namely, volatility of the working environment and the cooperation of earthmoving equipment. An overview is presented here for the robotic platform and its different modes of control in order to address these challenges.

2.1. Volatile environment

In association with global warming and other vulnerable factors, changing weather, extreme climate events as well as difficult conditions on a jobsite affect directly earthmoving processes. These influences have recently become a serious problem for earthmoving equipment and work safety in particular. This requires a synergetic effort from all stakeholders. Here, volatility implies varying, uncertain, unpredictable conditions, which may cause drastic changes in the operations on a jobsite. To take necessary measures and maintain earthwork performance and safety, more sensory and advisory information is required for the RAS-based stations and units used to control the equipment. A volatile environment may cause much more serious problems when it is also subject to harshness in earthmoving operations as earthworks usually take place under difficult conditions that have been widely referred to as 3D’s, namely Dull, Dirty, and Dangerous. On top of insurance and other remedial expenses, the increase in expenditure for a RAS solution to cope with a volatile environment has therefore imposed another dimension, that is, costs or Dear [8], because the money increases spent to cover the design, planning, and extra measures taking into account volatility factors. The budget limited in project procurement may necessitate a suitable solution of automation among the modes of control to leverage at a compromise.

Apart from 4D difficulties, earthmoving in some domains may also encounter disadvantages known as 3H’s, that is, Harsh, Hostile, and Hazardous, such as in a site with high levels of radiation, lack of oxygen, unhealthy level of pollution, high explosive risk, extreme temperatures and pressures, or in military earthmoving missions [2]. The application for RAS in such an environment offers a viable solution to satisfy the requirements of safety and resilience, whereby inputs from sensory systems, environmental logistics, and human–machine interfaces are utilized to realize interoperability, operator awareness, control systems, and autonomy oversight [9]. Harsh conditions may degrade the quality of collected data and performance of controllers when operating in field conditions. For reliable and resilient performance of RAS-based equipment for earthworks against environmental variations, disturbances, imperfect conditions, or ambient changes, the ubiquitous wireless sensing system should allow for a level of redundancy to achieve fault tolerance and system dependability.

2.2. Teaming capacity

In robotics research, the cooperative control of multiple platforms is among the emerging trends. In construction and mining automation, the cooperation of earthmoving machines, considered as a technological challenge, indeed, creates opportunities for research and development in robotics. To address this teaming capacity of autonomous platforms, RAS are playing the key role along with advanced technologies in AI, data science, IoT, and CPSs. A great deal of research and development in defense has been given to the management of fleets of cooperative platforms including ground vehicles. For the mining and construction sector, automation of multiplatform cooperation processes has also been a goal for Original Equipment Manufacturers (OEM). In a recent report [10], priority is taken in developing collaborative control strategies for construction equipment to carry out the work instructions in a digitalized framework. The aim is to connect all information on the jobsites with the equipment under intelligent machine control to substantially improve safety, environmental friendliness, and in particular, productivity.

It can be envisaged that an automated construction site in this decade will manifest the collaboration of bulldozers, graders, and excavators to undertake ground-based tasks without operators behind the cabin controls; in some cases, the machines will be without cabins. Unmanned aerial vehicles (UAVs) or drones in heterogeneous cooperation with ground equipment will play an important role in smart construction to provide platform-based and site-wide feedback to coordinate onsite platforms, including the possible access to volatile and harsh environments, operating in difficult and harsh conditions, surveying, inspection and monitoring, and most of material handling works [11].

2.3. Modes of control and RAS technologies

In this survey paper, the term “autonomous” is attributed to earthmoving machines that can operate without human operators (on board or via remote control) on the basis of a preprogrammed 2D or 3D production model of the job site. Levels of autonomy for unmanned systems can be classified by a framework based on mission complexity, environmental difficulty, and human independence [12]. The framework covers 10 scales from the lowest to the highest ranging from remote control to full/intelligent autonomy. In noncontextual autonomy potential, these scales can be rendered to nonautonomous, semiautonomous, autonomous, and fully autonomous levels. Although RAS applied to earthmoving tasks may appear to be technologically common across platforms, the level of automation exhibited by each platform can be different, depending on various factors. Focused directly on the platform-centric control process for earthmoving, the five modes of control (MoC) are proposed, namely, (i) functional assist, (ii) tele-operation, (iii) semiautonomy, (iv) full autonomy, and (v) cooperation [2].

Earthmoving tasks usually depend on the site conditions, and hence, their performance is often affected by changes in the work environment. In face of these changes, robotic platforms have been deployed to help maintain quality, productivity, efficiency, and safety [13]. With RAS applications, distinct advantages achieved can be attributed to the key automated functions of robotic systems as summarized in Fig 1, modified from [2] to include also AI and communication of higher generations for aggregating data and learning to continuously improve performance of complicated tasks such as multiplatform coordination.

Figure 1. RAS technologies for earthmoving automation [2].

The core RAS technologies cover following elements:

• • Perception, to extract and process sensing data from the machinery and jobsite;

• • Localization, to determine the dynamic position of the platform in the environment;

• • Navigation, to plan a safe path to move from an initial location to reach the final destination;

• • Control, to command its actuators to follow desired trajectories and perform the automated tasks from the lowest to higher levels;

• • Communication, to connect and update real-time information of internal and external states of equipment in networks with wireless IoT systems and learning schemes;

• • Coordination and networked robotics, to deploy the fleet of autonomous equipment working in a team.

3. RAS-based earthmoving in volatile environments

As earthmoving processes usually take place outdoor and hence, it is susceptible to changes in the environment, especially in a construction site with unexpected disturbances such as slippery terrains, extreme temperatures, impaired vision, electromagnetic radiations, and chemical erosion. The undesired impacts caused by these volatile conditions may include work accidents, damages, uncontrollable facilities and direct or indirect degradation of human performance, and project suspension. To deal with difficulty and harshness (4D’s – 3H’s) encountered at a work site, all modes of control are still being developed as per requirements from the market and industrial demands.

3.1. Localization and navigation

Although localization and navigation control are among the mature research for mobile robots and autonomous vehicles, their applications to earthmoving automation have to take into account unstructured and cluttered environment [14]. The weather volatile conditions may cause localization and navigation problems for platforms with Global Positioning System (GPS)-denied environments, featureless landscapes, loss of perception on obstacles as well as increasing difficult constraints in path planning. The relative and absolute positions of robotic platforms working onsite have been identified quite accurately by implementing real-time location systems (RTLS) or fusion of them, including GPS, IMU – Inertial Measurement Unit, LiDAR – Light Detection and Ranging, RFID – Radio Frequency Identification, UWB – Ultra-Wide Band, IoT-based sensors, encoder-based sensors, and cameras.

In the functional assist mode of control, GPS-based units can be attached to most earthmoving equipment for localization and navigation. Global navigation satellite system with 3D guidance system (3D-GNSS) can be integrated to robotic excavators for machine control and guidance system. To improve positioning accuracy, a technique called real-time kinematic (RTK) positioning has recently been used with a precision to the centimeter level [15]. This technique uses carrier-based ranging to provide ranges, and therefore absolute positions, which have a magnitude more precise than those available through code-based positioning methods. While high-precision dual-frequency GPS systems are available for earthmoving platforms, since GPS signals are affected by weather conditions, common processes such as digging, loading, hauling, compacting, grading, and paving performed by robotic platforms may not achieve the required accuracy under various weather events.

Unlike indoor localization having mostly fixed features and usually based on reliable design and recognition of visual landmarks [16], natural landscapes in the outdoor are drastically various and subject to volatile changes including lighting conditions, weather and seasonal influences, and human activities. Heterogeneous features of artificial landmarks are also uncertain. Therefore, operations of onsite robotic machines require context-aware and augmented SLAM (simultaneous localization and mapping) algorithms. In this regard, visual SLAM has more advantages in exploiting more detail information, for example, in forestry landscape with a sensor rig of RGB cameras, an IMU, and a GNSS receiver [17], or in urban scenery based on a heterogeneous landmark-based navigation approach with monocular vision [18]. Various SLAM algorithms have found a wide range of applications, including robot navigation and augmented reality (AR). It is of interest to use visual SLAM for reconstructing 3D color-mapped point clouds (collected from LiDARs or laser scanners) to reflect an uncertain terrain and identify onsite static and dynamic obstacles [19]. Visual SLAM is considered as efficient in a fast reconstruction of a dense 3D point cloud model of the earthwork site with an UAV platform combined with an embedded graphics processing unit [20]. The semantic SLAM combines 3D mapping and semantic segmentation to enable the robotic equipment to acquire information of a large-scale outdoor environment and facilitate reasoning ability for language-based human–robot interactions [21].

Localization in a GPS-denied environment is typically based on the sensor fusion approach to take advantage of both relative and absolute position measurements. In earth works, RFID, encoder, UWB, IMU, and IoT-based sensors are fused in relative positioning. The relative positioning method is widely used because it allows for high sampling rates and is inexpensive. However, it accumulates error over time if no appropriate compensation measure is taken. For this reason, an IMU is often used with intermittent absolute position measurements to reset the error after a certain period of time. The main advantage of absolute position inputs to an inertial navigation system is their independence from previous estimations, which tends to reduce accumulated errors. Its limitations include complexity of implementation and dependence on the environment structure. Besides, for low-cost solutions, the RFID technology [22] and IoT-enabled systems [23] can be attached on fixed objects, workers uniforms, and mobile equipment to enhance the proximity detection and safety assurance. Absolute positioning can be achieved in a GPS-denied environment via a map-matching technique. This approach uses geometric features of the environment to compute the location of the vehicle. The features can be lines and points that describe walls and corners or specific shapes such as rectangles or triangles depending on the object structure. For localization and navigation of earthmoving platforms, the commonly applied GPS can be alternated by the fusion of other approaches such as odometry, visual patterns, UWB, and the emerging deep learning neural networks for landmark detection [24] to improve accuracy and reliability.

For earthmoving automation, construction machines have predominantly utilized vision-based technologies, where cameras play an important role. For most surface earthworks or open-pit mining, where spaces allow for good lighting conditions, cameras remain an effective tool for localization and navigation with robotic vision. Indeed, for reconstructing 3D point cloud, stereo cameras having two or more lenses with a separate image sensor or film frame for each lens can achieve better efficiency than LiDAR regarding the postprocessing time, power supply, and data interfaces for scanning a terrain [25]. Thermal cameras, being able to recognize and capture different levels of infrared light, have an advantage for localization and navigation in the poor visibility condition. They help improve quality management of earthworks such as for nighttime operations such as to enhance the pavement durability [26]. Multispectral images, for example, obtained from RGB and thermal cameras and using semantic segmentation [27], can provide a better accuracy in real-time localization and navigation for autonomous vehicles operating under poor visibility or adverse weather conditions. A single camera using a complementary metal oxide semiconductor (CMOS) sensor can also be used for multispectral imaging to identify various features, especially when mounted on UAVs [28].

In underground mining, earthmoving equipment faces more severe 3H’s difficulties. Specifically, unclear landmarks, high humidity and temperature, communication ineffectiveness, GPS unavailability, muddy, and rough terrain inside tunnel cause equipment tasking unreliable, which degrades significantly the performance of tele-operation and camera-based tracking techniques [29]. Therefore, laser scanning technologies with enhanced precision techniques are in favor for underground localization and navigation. In ref. [30], a Generalized Iterative Closest Point (GICP)-based algorithm is applied to obtain a 3D point cloud of LiDAR-collected data to enhance SLAM resilience in confined spaces. Thermal camera can be fused with LiDAR sensor for autonomous exploration in underground mines [31]. A multisensor localization technique combining LiDAR and cameras has been recently developed to improve robustness of autonomous operation within two operational Australian underground hard-rock mines [32]. Toward full autonomy of equipment in overcoming such hazards as rockbursts, squeezing and creeping rockwalls, ditches of water drainage, as well as airborne dust, the extrication of humans from underground mines to enable improved safety and efficiencies still remains the top priority.

3.2. Sensing and perception

Sensing and perception in RAS-based earthmoving aim to develop and deploy accurate measurement systems for real-time monitoring of dynamical parameters inside and surrounding the equipment (e.g., position, speed, acceleration, location, images, temperature, humidity, moisture, pressure, frequency, etc.) as addressing the state identification problem [33]. The collected information is essential to feed to a sensor-based controller to adapt any changes of environment and increase the contextual awareness in different and complex tasks at jobsites [34]. For instance, a machine guidance system for a bulldozer was developed [35], based on the fusion of sensors such as IMU and RTK-GPS, to provide navigation assistance and to increase earthwork productivity via estimation of the blade pose. Besides, machine’s pose estimation [36] is needed to maintain performance during weather events by enhancing perception with multiple sensors attached on platforms. This technology requires the advanced methodologies to fuse the internal parameters and external information for the relative and absolute position and orientation of the platforms and its attachment. In dealing with environmental volatility, the estimated poses of heavy equipment are critical not only to reduce the damage of collision but also to increase the precision of cooperation and autonomous level. The integration of multiple sensors such as factory-installed GNSS antenna and receiver, IMU or enhanced IMU, and hydraulic cylinder stroke sensors on an equipment can constitute an intelligent sensing and controlling framework [37], tele-operation relies mostly on quality of wireless communication system for safe operation.

In the tele-operation mode of control, vision-based and auditory sensors increase the environmental perception capacity with the use of multiple cameras for streaming live videos from the sites to the control center. Since the regular camera provides 2D images, information obtained is insufficient to cope with uncertain position and orientation of complex and high degree of freedom actuators, in particular in volatile weather conditions. Accordingly, the 3D pose estimation by using stereo vision and its fusion with other RTLS offers a prevalent approach [36]. Recently, 3D point cloud images reconstructed by the laser scanning methods to provide reliable data have been also promising because this technology is less vulnerable to operational conditions, for example, sunlight, darkness, or harshness on the field [1].

As safe and efficient tele-operation requires a deep perception of site environment through a variety of sensing devices, performance of remote monitoring and control relies on network communication and data throughput. However, the communication speed and connectivity are subject to field conditions, and hence, directly affecting the quality of service (QoS). Therefore, the specialized design of communication systems at construction sites and underground mines need to consider not only the reliability, speed, and coverage of networks but also security and effects caused by long-distance tele-operation [29]. These may include jitters, data packet loss, and blackout but most serious is time delay. To mitigate tele-robotic time delay, control approaches have been proposed, covering model-based, adaptive-based to predictive control techniques. Recent prediction approaches based on machine learning and neural networks are promising in addressing time delay to this open problem [38].

3.3. Control and planning

For automated earthmoving, to improve the ultimate work capacity and task efficiency on a jobsite in face of environmental volatility, research and development on modeling, control, and planning remain active. For ground-based mobile machines, specific difficulties caused by harsh environments, weather changes, or dynamic conditions may include slippage due to heavy rain, snow, flash flood, or operations on various ground types like rock, sand, mud, soft soil, and uneven terrain.

For autonomous excavation, in addition to known compliant control strategies like impedance or admittance control, the task planning and control can also be integrated in generation of dig cycles regardless of the soil composition by using a bucket tip force–torque trajectory. The bucket forces and motion of the excavator arm are controlled using hierarchical optimization along with a large-scale iterative planner for consecutive execution of single digs to form the desired in-ground geometry. This strategy can handle different types of soil to deal with the tool–soil interaction for different terrains using the Vortex simulation package [39]. Field validation of an iterative learning-based control algorithm for autonomous excavation has recently been reported in ref. [40] with a 14-ton capacity load–haul–dump (LHD) machine for two different types of excavation materials: fragmented rock and gravel. With iterative learning, the admittance control parameters can be automatically updated based on the fill weight error at the end of each excavation pass.

Since possible incidents due to unpredictable changes in the environment can take place on a jobsite, earthmoving equipment should adopt appropriate measures based on environmental sensing. As such, a RAS-based controller has to be equipped with the capability of observing the work environment and responding to its changes. Recently, an architecture combining perception using LiDAR and cameras, along with learning-based and optimization-based planning has been developed for a 7.5-ton excavator to perform autonomous tasks of material loadings continuously in 24 h [41]. In industry, some OEM have incorporated the environmental perception function in their remote controllers, particularly for dozers and loaders [42] or implemented RemoteTask systems for skid-steer, multi-terrain, and compact track loaders, where the machines can be controlled at a safe distance as far away as 300 m in dealing with difficult environments. An automated bucket-loading system, the Autodig, has been successfully applied to improve bucket fill and cycle times to boost productivity in mining and construction, with a high tolerance of uncertainties in the soil and working conditions. To be able to handle a vast amount of data coming out from the adaptation to changes, the 5G mobile communication technology has been recently applied to not only significantly reduce communication latency [43] but also support data processing for robust systems of connected sensors for weather-related events to mitigate incidents on a jobsite [44].

As previously mentioned, the smart construction framework could enable the IoT connection for an earthwork site and its 3D visualization. The service features include site topography and design, information of area, shape and volume, and especially the intelligent machine control of earthmoving equipment. In terms of control, the intelligent machine controlled (iMC) technology developed by Komatsu for their dozers and excavators is well integrated with 3D machine control components at the factory level for sensing stroke, load, and other autonomous actuations based around the 3D GNSS and IMU as well as its enhanced version (IMU+) [37]. The iMC system allows for the RAS-based equipment in dealing with difficulties arising from changes in the work environment for various earthwork types to ensure accuracy and efficiency.

3.4. Skid-steering and slippage

For mobile robots and autonomous vehicles, the integration of highly accurate sensors and their fusion are useful for navigation control. However, slippage remains one of the main issues associated with platforms navigating on a jobsite particularly by skid-steering. Slippage is usually associated with skid-steered platforms in robotic earthmoving, wherein high torques are required to perform a curvilinear motion. As a consequence of slippage, those platforms may be unable to follow trajectories generated by the motion planners. To incorporate this issue, dynamic models can be developed to combine the machine’s wheel or track slip with mechanics of the terrain. The models can be obtained by using estimation algorithms such as a system observer or an extended Kalman filter (EKF). For modeling and calibration dealing with skid-steered platforms, an online leaning framework has been proposed by combination of the slip and terramechanics models of wheel–terrain interaction using Kalman filtering (for kinematics updating) and neural networks (for dynamics updating) to achieve the high performance in curvilinear motion with improved energy efficiency [45].

For the wheeled mobile robots, slippage may occur to the wheel’s effective radius variation due to heavy load, tire wear issue, and an extra component from a significant change in the slip model parameters under extreme weather conditions. For example, heavy rain was the cause of a collision between two autonomous trucks [7], when the unloaded truck traveling at about 14 km/h lost traction and slid to the other, loaded and traveling at 27 km/h. Mobile wheeled platforms, such as front-end loaders, tractorshovels, pay-loaders, high lifts, and skip loaders, are all subject to slippage on snowy or wet roads and rough terrain, causing the tires to wear out faster and leading to less tractive force. Passively, the torque proportional differential can be replaced by the limited slip differential to deliver effective driving force to both wheels for enhanced grip and less slippage during travel. Actively, RAS-based traction control can be developed to deliver the highest possible tractive force by using sensors’ data and mathematical modeling of the tire–ground interaction [46].

For tracked vehicles, longitudinal and lateral slippage occurring during straight maneuvering can be derived by approximating the terrain-mechanics slip model or using a transient shear model for the slip and track contact on hard ground [47]. In ref. [48], a slip model based on the instantaneous centers of rotation is developed based on position measurements and an EKF. The proposed model was validated in experiments on a 13.6 ton tracked platform equipped with a RTK GPS. Tracked slippage may cause inaccuracy in the platform localization, especially when traveling on loose slope. In ref. [49], slip-compensated odometry, which applies the slip model to the kinematics of a skid-steering vehicle, is proposed for evaluation of the slip estimation of the interaction between a tracked vehicle and an unknown terrain. Although evaluation of the slip model and slip-compensated optometry proposed therein was conducted on a small tracked vehicle on an indoor sandy slope, the promising results can be extended to a robotic excavator or dozer with skid steering. For platform’s turning, longitudinal slippage can be derived from an empirical equation for the relationship between slip ratio and input velocity, and lateral slippage is obtained from a regression function.

3.5. Tool–soil interaction

In robotic earthmoving, an automatic control system developed for executing an earthwork task (digging, bunding, tilting, trenching, profiling, or grading) should take into account of the tool–soil interaction forces arising from the contact in real time. They represent reactions to the driving tool and vary in accordance with the soil properties characterized by the soil type, humidity, and other terramechanic parameters [50]. Thus, from the interaction in the contact phase of the machine’s attachment, the digging forces are highly nonlinear and depend on the soil composition (sand, rock, gravel, mud, etc., or a mixture of them) and its properties, soil volume, tool shape, and working cycle. Interactions between equipment tools and soil are important for autonomous operation of an excavator. Small variations in soil properties may cause significant changes in its static and dynamic behavior, particularly in an inhomogeneous medium. This explains why weather changes or other volatile factors can affect directly work productivity of the earthmoving process.

In the soil cutting and pushing process, the edge of the end effector first penetrates the soil up to a certain depth and the machine then starts cutting, shearing, and pushing the soil. During this interaction, the blade experiences enormous resistance owing to friction, cohesion, and adhesion between the tool and soil, and the soil and ground. The forces acting on the tool vary in a complicated manner that may negatively impact the equipment performance. In the literature, the resistance or “draft force problem” has been tackled either experimentally, or by developing analytical models and using numerical methods [51]. Methods used for parameter estimation in different tool–soil interaction models have included multilayer neural networks and swarm intelligence [52], discrete element modeling [53], the finite element [50], and the spectral element method [54].

Soil parameter estimation enables prediction of interaction forces by changing tool parameters such as the depth of cut, the attack angle of the tool, and digging strategies [55]. Recent effort has incorporated the soil–tool interaction also in designing an optimal bucket to achieve the highest productivity of the tool and the platform as a whole [56]. However, to achieve accuracy in modeling the soil behavior, more complex rheological models are required, often at a larger computational time and hence may be infeasible for application in a real-time control. Therefore, in addition to the effective parametrization of the soil–tool interaction model in dynamics of the control process in real time, another approach is to treat it as a disturbance and design a robust strategy for control.

As the distribution of pressure and force is nonuniform when the equipment attachment tips interact with different soils and loads, tactile sensors using the piezoresistive, piezoelectric, optical, capacitive, or elastoresistive effect can provide measurements of the medium reaction with high accuracy for estimating soil parameters or predicting tool wear. Information of the terrain chracteristics from tactile sensing can assist with machine traversability on different terrain types to improve navigation on earthworks sites [57]. In ref. [58], soil pressure was measured with 10 tactile sensors attached on the plough. Results obtained are useful for validating numerical models of the soil cutting process leading to increase task efficiency. By merging force sensing (1D) with LiDAR intensity information (1D) and other proprioceptive and exteroceptive sensors for 3D ground map, the 5D map generated can be used to estimate the bucket-load volume and effectively monitor the excavation progress [59].

Under the influence of a volatile environment, the tool–soil interaction would become more complicated and unpredictable. Remote sensing methodologies could therefore provide dynamic information [60] to compensate for such variations, and robust controllers could mitigate the disturbances impact. Solutions to the tool–soil problem in earthmoving often come from control strategies at both low level and high level. For example, in robotic excavation, a digging strategy can be rendered to track the piston position and ram force of each hydraulic cylinder for the axis control of the boom, arm, and bucket. The idea can be elaborated in compliant control schemes such as impedance or admittance control, to adjust the dynamic relationship between the end effector position and the exerting force, considering the soil contact. Overall, to achieve the effectiveness of automated earthmoving, a control strategy has to deal with soil parameter identification [61] and force prediction [62]. In this regard, the control problem considering the tool soil contact for earthmoving under volatile conditions remains open for further research in both academia and industry.

3.6. RAS-enabled functions

Since earthmoving often involves dull tasks such as compacting, loading, and dumping materials, RAS-based machines can take over the repetitive and tedious operations to reduce the labor and time expenses. In road construction, a grader can be equipped with a functional assist unit to ease the operator’s control of the machine’s blade and ripper functions in order to achieve high performance of grading on various difficult environments and hence to improve its productivity. For example, the equipment reported in ref. [63] can handle such functions as Grade control, TOPCON 3D machine control (3DMC) plug-and-play, and rearview monitoring. Especially in dealing with environment changes, the grader can have a full-color high-resolution monitor with Ecology Guidance, as well as pioneering KOMTRAX telematics system and monitor to provide machine metrics, fuel consumption, and plus performance information.

Indeed, to cope with the difficulties arising from uncertain and unpredictable factors of the environment, RAS-based technologies have been developed to provide assist functions on equipment used in the earthmoving industry. As introduced in ref. [64], a versatile supportive and advisory system can offer multiple productive services. These cover common functions for earthworks, as illustrated in Fig. 2, including:

Figure 2. Functional assist system on Volvo equipment [64].

• • Load Assist: to optimize load cycles via a set of smart apps for boosting the efficiency with such features as (i) a dynamic load weighing system for the bucket’s load to enable safe operation and avoid overloading, underloading, reweighing as well as waiting times and (ii) real-time guidance to operators in actions and improvement of their operation technique.

• • Compact Assist: to provide in real time all onsite information necessary for soil or asphalt compaction such as compactor’s pass, temperature mapping, and density calculation to ensure consistent quality and avoid wasteful over-compaction.

• • Dig Assist: to improve accuracy and efficiency in all excavation tasks such as digging, leveling, trenching, grading, or profiling from an in-cab tablet. The Active Control system can assist in controlling motion of the boom, arm, bucket, swing, bucket’s tilt, and attack angles, as well as limiting the machine’s swing, the depth and height of the cut. The Co-Pilot system can assist in locating the construction equipment to centimeter accuracy with sensors and the RTK and GNSS, as well as importing external 3D design plans in complex projects.

• • Haul Assist: to optimize haul cycles with on-board weighing and onsite monitoring map for hauler traffic in real time.

• • Pave Assist: to ease with the heavy work of paving in a harsh environment with a Material Manager identifying the machine throughput efficiency and keeping a record of calculated distance and tonnage paved, paved area and CO $_2$ emissions, the volume of delivered asphalt and its thermal profile, as well as the weather view.

• • Report Assist: to provide comprehensive reports on productivity, fuel efficiency by using telematics data for improving the efficiency and uptime of the fleet, and thus, maximizing the task capacity and work efficiency of the machines’ operations.

Those functions can help an operator dealing effectively in one way or another with difficulties caused by a volatile environment. For example, as changes in the soil mechanic properties may affect loading or hauling, real-time guiding advice and automated weighing information can help an operator adjust or give input to control strategies. Likewise, various excavation tasks in face of unpredictable conditions can be adjusted with the Volvo Dig Assist service from Co-Pilot system. Moreover, information of environmental parameters, weather conditions along with machine characteristics can also be collected on-board and even available for reporting purposes.

In addition, Volvo also provides its Simulators for training on authentic scenarios to manage skills and to support evaluation and development with virtual data analysis for monitoring project progress, especially with any presumed weather conditions in dealing with volatile environments. Similar products are also available in mining and rock excavation, including the Automine, an automated loading and hauling system for underground hard-rock mining, and data-driven advanced analytic systems for improving maintenance, safety, productivity, and operational services of mining and rock excavation equipment [65]. Indeed, the use of remote monitoring integrated with advanced analytics and cognitive technologies for fusion of equipment data from a range of resources can automatically analyze the patterns to increase mining performance and decrease the cost per tonne with RAS-based and information technologies. Environmental awareness from onsite data measurements and analytics can effectively assist a tele-operator to manage in real time the jobsite under varying work conditions.

In dealing with volatile environments, it is essential to quantify the weather-related impact, tracking all aspects of the machine, earthworks, and the environment in accordance with automated capturing and predicting weather profiles. Field management for robust earthmoving may rely on multiple third-party weather services such as World Weather Online’s weather API (Application Programming Interface), AccuWeather, OpenWeatherMap, Weather Underground, and Weather Forecast from the local Environmental Protection Authority (EPA). These services can be integrated via APIs to allow RAS-based platforms to take necessary corrective actions either by controlling the equipment at the low level of actuators, job guiding, task re-planning or rescheduling at the higher level, and safety/difficulty reporting. The Weather View in multiple productivity services mentioned as part of the various functional assists in Volvo Co-Pilot system, as illustrated in Fig. 3 for paving tasks, can provide up-to-date information for better planning and adapting with earthworks against weather events.

Figure 3. Co-Pilot system featuring (a) thermal profiling, (b) weather view, (c) material manager, and (d) jobsite view [66].

In terms of handling weather impacts, it is promising to explore frameworks of IoT-enabled collocated low-cost sensing networks, such as the one used for monitoring urban particulate matter emissions on a construction site [67]. This approach can offer an inexpensive solution to provide microclimate information, to gather trusted monitoring data, and forecast meteorological trends for a local site for earth works. With increasing demands for automation in earthmoving for the construction and mining sector, the research and development for application of RAS-based technologies remain to be active in the next decade, given changes in climate and the environment.

4. RAS-based multiplatform teaming and machinery collaboration

Along with the presence of mega construction projects as well as underground and surface mining sites around the world to address various demands of urbanization and economic development, earthmoving processes also need to satisfy specific requirements, for which the cooperation of multiple fleets of equipment is essential to meet the expectation. In this regard, RAS technologies can be applied to achieve the goal of teaming of homogeneous and heterogeneous platforms. Indeed, joint operation of machines can help deliver many demanding tasks that a single machine cannot achieve alone with constraints on time and quality. This is obvious since a multiple-robot system can accomplish certain tasks that a single robot barely can, or cannot at all, considering work capacity and task efficiency. To this end, the study of multiple-robot systems can naturally be extended from research on single-robot systems. To facilitate the coordination of multiple machines on a jobsite and monitor the progress, it is essential to reliably gather information of the site and equipment as well as to effectively control the cooperation of each machine in the team. As such, on an earthmoving site, the autonomous operations of construction equipment in a group requires the automated machine member to have the ability to form certain patterns of navigation together, interact with each other when performing a shared task, autonomously decompose a complex task among others, as well as effectively manage and prioritize events.

In earthmoving automation, the automatic task allocation and planning between each construction equipment in a team as well as cooperative control of the team are becoming increasingly dependent on distributed control, connectivity and data exchanges between them. The teaming capability of construction equipment has been explored in civil industry and defense. For example, RAS-based cooperation between an excavator and a haulage truck has successfully been demonstrated in the Terraforming Heavy Outdoor Robot (THOR) project [68] or, more recently, the cooperative control of multi-swarms of aerial, ground and marine unmanned vehicles [69] has been conceptualized in surveillance for early detection of escapers from a restricted area. Here, the focus is on automated earthworks in construction and mining. Techniques to be used for their coordination and cooperative control for equipment used in earthmoving are briefly discussed in this Section.

4.1. Multi-agent systems

Multi-agent systems (MASs) are computerized systems or machines that comprise multiple interacting intelligent agents. Given their autonomous, cooperative, and learning attributes, MAS are suitable to address the essential problems of “collaboration and consensus” among stakeholders in construction projects. This can be explained by the three prominent attributes of MAS, namely (i) autonomy: the ability of sensing, self-decision and response of each agent in the system; (ii) cooperation: agents collaborating to gain an objective that an individual one could not acquire; (iii) learning: agents self-evolved, adapting with environment and enhancing performance. In addition, taking advantages from software engineering and AI, the MAS framework facilitates the system capability of dynamic planning and scheduling to reduce project’s cost, completion time, and resources.

With the dynamic environment of an earthmoving site scattering with material, machines, wastes and humans, planning of mobile platforms can be achieved from a MAS-based solution for a consensus problem [70]. By considering machines as agents, MAS may be useful for collision avoidance between RAS-based platforms. Indeed, collisions between machine–machine or machine–obstacle remain a challenging issue for safety of autonomous cearthmoving. As mentioned above, a collision of two tele-operated haul trucks on a mine at Pilbara, Western Australia occurred due to a communication drop-out from heavy rains [7]. This indicates the need of considering volatile environment in a RAS-based research and development for the control and communication of cooperative machines. In this regard, for multi-robot systems, a generic method, called $\epsilon$ -Cooperative Collision Avoidance ( $\epsilon$ CCA), has been proposed in ref. [71], based on an optimization in the space of control velocities in which moving obstacles are decision-making agents. Soft computing approaches are also promising for MAS as preliminary reported in ref. [72] based on human’s inner attention mechanism as a communication and multi-modal information fusion strategy, or in ref. [69] based on coevolutionary genetic algorithm for heterogeneous teams of multiple uninhabited vehicles.

4.2. Cooperative mapping and formation control

Mapping is the process of recreating the virtual environment for autonomous machinery to operate and interact with the real-world environment from its sensing information. The recent development of swarm robotics has expanded to an architecture known as multi-robot simultaneous localization and mapping (multi-SLAM) [73]. While the efficiency and processing time can be improved, implementation of the architecture is restricted due to constraints such as communication bandwidth, memory requirements and problems faced during map merging and co-ordinate transformation. In construction, where different machines can share the work on a jobsite, various platforms can also cooperate automatically using RAS technologies to fulfill complex tasks. For this, a promising hierarchical architecture for a group of heterogeneous mobile robots was proposed in ref. [74] using information space and sensor models. The algorithm can fuse limited sensory information, interrelate dissimilar data obtained by individual heterogeneous robots, and allocate various exploratory tasks to each of them in order to complete the environment map.

To obtain topographic information of earthwork sites, UAVs can be used to capture high-resolution images that can be used for 3D reconstruction. To improve model accuracy, the fusion of photogrammetry, laser-scanning, GPS and other sensory data can create a better point cloud map [75]. One advantage in using UAVs is that they can cover obstructed or inaccessible areas where ground vehicles cannot reach. The heterogeneous combination of ground vehicles and UAVs can enhance the real-time tracking of machine locations and trajectories. For instance, multiple-session mapping and onsite object tracking with Visual SLAM using UAVs can extend the maps with fast reconstruction of the 3D point cloud [20]. From these merits, drone 3D mapping has been included in Komatsu’s smart construction framework [76] integrating with IoT and 3D visualized technologies to capture the 3D point cloud data and map the updated earthmoving sites, which are inputs of multiple tasks such as progress tracking, stockpile volume managing, online collaborating and communicating between the autonomous machines.

The control of unmanned vehicle teams has been an active topic in robotics research. This involves the initialization, establishment and maintenance of a group of mobile robots moving in a desired geometrical shape or formation. It is known that formations of automated vehicles can been applied in intelligent transport systems to save energy and enhance safety. Formation control has attracted many researchers with majority of works focusing on the leader-following methodology based on behaviors or models of multiple robots. In ref. [77], a decentralized behavior-based algorithm has been proposed where the robust formation is achieved by maintaining the distance and angle of each robot toward the leader robot without information of its location. To avoid collisions, the heading angles of all robots are determined by generating an escape path along which a robot can move toward a safe layer. The leader follower technique in robotic formation was used to form a convoy of trucks equipped with a laser-based sensor (LiDAR) used for maintaining the distance clearance and demonstrating the V2I concept [78]. One of the key issues in formation maintenance and path planning is to resolve conflictory constraints between the leader and the followers, for which the optimization and algebraic graph theory frameworks [79], the Nash–Stackelberg equilibrium in game theory [80] and particle swarm optimization-based algorithms [81] offer promising solutions.

Teaming technologies for autonomous earthmoving equipment are evident as can be seen in autonomous loading by a front-end loader or an excavator and a dump truck, for example, [68]. Indeed, the cooperative operations of equipment obviously add to improved task efficiency on an earthmoving site, giving more room for effort to address requirements on sustainable development. Therefore, this theme is being pursued by leading equipment manufacturers. Figure 4 shows a team of Caterpillar equipment, autonomous dump trucks, surface shovel, and bulldozer working together on a surface mine. Semiautonomous cooperative control technologies are expected to reach higher levels of maturity in the future, given the need of machine coordination in smart mining, as reported by Komatsu, which has been recognized as international leaders of managing climate change [10].

Figure 4. Caterpillar autonomous equipment teaming on a surface mine [82].

The growing interest in driverless multi-vehicle networks has resulted in may research works focusing on estimation, perception, planning, and control to support cooperative and coordinated vehicle autonomy. In construction and mining, the coordination and distributed control of heterogeneous machines on a jobsite have stimulated many OEM in applying advanced technologies in robotics and automation, sensing and communication as well as data analytic and AI to be able to deploy a fleet of autonomous platforms to share a complicated or iterative task.

4.3. Cyberphysical systems, communication, and AI

The concept of integrating communication, control, and computing for systems to operate in real-time along with physical machines in feedback loops, whereby physical processes interact with computations to maintain work performance in face of the changes of environment is known as CPSs. It represents a holistic architecture of virtual models, hardware (equipment, sensors, and actuators), and a communication paradigm such as the cloud-based IoT to deal with coordinating and managing large-scale projects. Virtual objects can be generated from Building Information Modeling (BIM), which is a set of new processes to design, plan, and construct buildings [83].

For earthmoving, a cloud-based CPS framework can consist of independent and heterogeneous components (e.g., sensors, actuators, gateways, and server) and dynamic composition of their functionalities, along with smart nodes mounted on various equipment to monitor wirelessly their operation. The framework can allow for remote management of multiple big projects simultaneously in different locations, for example, highway construction and photovoltaic power plant construction. It consists of two main parts, an onsite IoT network and a cloud-based system with user interface for the project manager [84]. In volatile conditions, the IoT can support communication, sensing, and control for platforms with onsite digital intelligence. As an example, the internal IoT-based sensors automatically detect and send alert signals related to equipment states for active maintenance and replacement before any problem could occur. In combination with RFID technology in asset management, the transportation of material between different sites can be updated in real time, and the response of operators during logistic is also monitored to mitigate potential collisions or unexpected events [85]. To facilitate the teaming capacity, the machines make themselves recognizable, aggregate information, and share data with others to perform automated tasks. As such, the whole system can enhance reliability and dependability to cope with changes in the environment.

Multiplatform teaming would require system-level technologies to handle to the complex interaction between earthmoving equipment, workers, and materials or objects in cooperative works [1]. Stand-alone activities of each RAS-integrated platform of the team would need to efficiently fulfill the task decomposition in earthwork coordination. Hence, the context awareness of autonomous machines on the site needs to be secured for harmonious cooperation to improve productivity and avoid the unexpected collision and accident. For this, the wireless communication between the equipment based on IoT networks is increasingly adopted to schedule and manage the cycle operation of equipment [34] owing to advantages from 5G technologies [86]. To achieve error-free wireless communications for cooperation of group of equipment, the intelligent excavation system (IES) proposed in [87] suggests the use of multiple hardware and software modules which interact each other with diversity of communication protocols (e.g., TCP/IP, SCI, etc.). This framework is managed by a data communication manager and an equipment data management system to coordinate large amount of data from micro (sensing and control) to macro (task planing) levels.

Robustness of 5G networks for tele-operated cooperation is evident as experimented in the wireless cooperative control of two platforms (one backhoe and one crawler dump truck, or one excavator and one front loader) [88]. By using a 5G high-speed, high-capacity, and low-latency system, the onsite images of eight cameras with 8K resolutions including the sound have been streamed in real time over the distance up to 800 km to provide the operators with true feelings of the “on-field” condition, as illustrated in Fig. 5. The feasibility of wireless cooperative control of autonomous platforms has confirmed the possibility of IoT-based control framework under the integration of 5G network for RAS-based multiplatform coordination.

Figure 5. Remote control in cooperative manner with 5G technology [88].

For performance monitoring in autonomous operations of various earthmoving platforms, it is essential to accurately identify their activities on a site. To this end, vision-based systems are useful in extracting information from captured images, particularly when considering the interactive operations among equipment. An activity identification framework was proposed in ref. [89] to cover interactive aspects of earthmoving operations, including equipment tracking, action recognition of individual equipment, interaction analysis, and postprocessing. Typical interactions of excavators and dump trucks in earthmoving processed from thousands of image frames from actual construction sites could be used in analysis of automated operations of earthmoving platforms. Since the wireless information aggregated from the fusion of sensors, machine controls and network communications produce a huge amount of data, AI techniques such as the convolutional neural network (CNN) [89], or a deep learning algorithm [90] have been applied for processing, evaluating, validating, and predicting to eventually result in suitable control strategies for online cooperation.

Recent developments of machine learning and deep learning techniques have found many applications in construction automation such as equipment tracking or condition monitoring. As earthworks often involve cyclic operations, it is in favor for model training to learn repetitive features from collected data using vision-based systems. In ref. [91], a tracking-learning-detection algorithm is developed using a hybrid deep learning model with CNN and long short-term memory recurrent neural network (RNN) to automatically localize vehicles and recognize the sequential actions of robotic excavators such as digging, hauling, dumping, and swinging. These results are promising to be integrated into a cooperative framework to predict the next steps of interaction between heterogeneous platforms. To improve tracking accuracy, for example, in the nighttime or in poor lighting conditions, a deep learning illumination algorithm CNN-based feature extractor has been proposed in ref. [92] for data assimilation and object identification in facing the challenge of illumination variations and motion blurs [92]. Feature preservation capability of deep learning is also enhanced with a hierarchical CNN algorithm with autotuned thresholding for crack detection in built infrastructure [93]. Other applications of deep learning include also construction work management, sewer assessment, and 3D point cloud mapping [94], or tele-robotic time delay mitigation [38]. However, for training and applying effectively learning-based frameworks, underlying limitations, such as videos and images captured in different viewpoints, lighting conditions, multispectrum, and volatile weather, have to be resolved such as big data requirements for model training in general earthmoving conditions. Also, intensive computation for real-time processing and controlling high number of equipment, prior experiments for various building styles or locations should be take into account [95].

The teaming of multiplatforms on a site can be decentralized to leverage the performance of a cooperative policy in any scale of swarms via reinforcement learning [96]. Moreover, as teaming can be combined with other modes of control, interactions with human workers are also needed for the fleet cooperation in some scenarios, where a higher safety level should be ensured [97]. Towards a new generation of intelligent earthmoving equipment, open and interoperable frameworks using AI have been developed for retrofitting to existing fleet of heavy construction machines [98] to increase the equipment’s awareness to the environment and their ability to make decisions on coopervative tasks with other platforms in the team. These intelligent machines are aimed at acquiring the ability to “perceive their surroundings and be continuously alert, helping operators work more efficiently and safely” [99], as conceptually depicted in Fig. 6.

Figure 6. Safe and efficient teaming of using embedded AI [99].

With disruptive technologies including CPS, 5G communication, and AI, teaming capacity in earthmoving automation is expected to be enhanced to achieve full autonomy on an earthmoving site. Embracing this vision, the Intracore Concept-X, has been launched as a versatile solution for autonomous earthworks with the ultimate aim of increasing productivity and safety, expected to be commercialized by 2025 [43]. The work site topography will be surveyed via 3D drone scanning, whereupon 5G-based remote control technology will be deployed to operate RAS-based equipment such as excavators, wheel loaders, and haul trucks, as illustrated in Fig. 7.

Figure 7. Teaming of autonomous machines in Concept-X [43].

5. Safety, task efficiency, and work capacity: Industrial perspectives

Most of the earthworks in construction and mining are falling in a category of 4D’s and 3H’s tasks. Fortunately, they can be automated to a certain level, depending on complexity. In this section, we look at the direct advantages of RAS applications to the earthmoving industry from recent achievements and development strategies of the OEM in addressing the challenges mentioned above. The ultimate goals for automation here are for RAS-based earthmoving equipment (i) to improve the safety by expanding the work capacity to replace or assist human operators in all functions, mitigating any risk and (ii) to improve the productivity from increasing the task efficiency by acquiring more information of the machines and environment to apply suitable control and coordination strategies while reducing the operational costs. Overwhelming evidence from OEM, construction, and mining industries around the word has verified the significant contributions made by RAS technologies in improving productivity, safety, efficiency, and operational cost-effectiveness, particularly earthmoving equipment. The development of RAS-based measures to address the topics of environmental volatility and platform teaming can be considered as directly contributing to improve safety, efficiency, and productivity of earthmoving. This can be broadly grouped into:

• • Functional assist systems for monitoring and control, wherein RAS-based technologies – from cameras, thermal imaging to self-aware machinery, from information gathering and control implementation to weather prediction and onsite digital report and analysis – are able to monitor equipment operations, guide predictive maintenance, and support automation of earthmoving tasks to achieve higher productivity.

• • Functional assist systems for sensing and localization, wherein RAS-based geographic information systems (GISs) and positioning systems – GPS, GNSS, and location-based services – can be applied for all earthmoving tasks, and importantly, reinforcing safe operations of a wide range of mobile equipment from single platforms to entire fleets,

• • Semiautonomous and remote control of heavy equipment for almost all types RAS-based earthworks – crushing, rock breaking, shovel swing loading, drilling, tunnel boring, and blasting,

• • Autonomous haul trucks and loaders for which RAS-based driverless technology can substantially improve productivity and task efficiency – about 15–20% increase in output, 10–15% decrease in fuel consumption, and 8% decrease in maintenance costs [100].

In terms of human safety, the earthmoving industry has indicated that RAS-based tele-operation and semiautonomous operation can improve work capacity of the equipment, and hence, reduce time to commute from the office to the field, decrease the exposure of operators to the harsh onsite environment, relieve the ergonomic strain under long hours working inside the carbine of vehicles, remove the need for bystanders to perform grade checks, as well as enable virtual fences to be set when operating in confined environments [29]. Besides, less being contacted with hazardous conditions such as underground or in the mining tunnels with toxic gases can be applied to earthmoving equipment to improve both productivity and safety. The semiautonomous tractor system (SATS) installed for mining machines has several safety functions and been equipped with proximity detection and collision awareness systems as well as emergency autonomous stop devices (A-stops) for machines and operators, following a collision at Peabody’s Wilpinjong Coal Mine between a SATS Caterpillar dozer and a manned Hitachi hydraulic excavator from poor sightlines and communications breakdown [101].

Offering a broad selection of products which are suitable for a wide range of applications, combination of innovation, and various technological improvements, Caterpillar has been a key player in the use of RAS-based application to heavy equipment in earthmoving. Indeed, according to the company’s 2017 comparison report [102], capacity and efficiency in earthmoving phase have been significantly improved by 30% to 40% thanks to the application of technology in the information and communications technology (ICT) and RAS-based functional assist mode. The earthworks involved in the analysis cover grading, ditching, moving embankment material, and compacting with and without the use of GPS machine control and guidance, machine drive power with intelligent compaction and mapping, smart payload as well as progress monitoring systems. This has resulted in increasing productivity and corresponding reduction in machine hours, reduced fuel consumption, while enhancing safety and resource availability as requiring less people on the ground.

In recognition of the important role of RAS technologies, Komatsu has invested in smart construction domain with the expectation to increase productivity, machines cooperation, and progress of project during the construction phase. In the 2019 report, the OEM presented the vision on digital transformation of the construction industry through five levels of optimization with automation and optimization as the top two levels, emphasizing the role of automated earthmoving [10]. Specifically, safety and productivity are expected to be improved through increased automation using additional Retrofit Kits for existing equipment, and then with autonomy being achieved via communication and coordination among the machines. The future will witness the collaboration of all equipment to carry out the work instructions issued by the digital platform using sophisticated automation and autonomy technologies, for example, with ultrahigh-speed mobile communication systems and high-precision global navigation satellite system (GNSS).

RAS-based effort from Hitachi Construction Machinery has focused on the semiautonomous mode of control for continuously increasing productivity and efficiency of earthwork tasks, as well as adopting innovative technologies. The OEM has notably invested in a new generation of its heavy equipment to achieve green characteristics for an eco-friendly construction site. Innovations such as hybrid battery-powered excavators and hydraulic systems ensure energy-efficient solutions for a wide range of construction processes including earthmoving. At Bauma 2019, it showcased a range of construction equipment that can be factory-modified to meet the demands of specific activities, such as slope finishing and deep excavation. Of interest is the excavator equipped with ICT and RAS-based machine control [103] that can complete excavation tasks 50% faster than a standard excavator, facilitated by the semiautomatic operation of the boom and arm with overcut protection and a higher accuracy using a 3D vision system.

For productivity enhancement, Volvo Construction Equipment (Volvo CE) has demonstrated the prototype for autonomous wheel loaders which could operate unmanned and reach the high productivity of 70% as much as a skilled operator’s level when loading, unloading in much higher safety level during a repeated cycle for one hour, and 45% faster than traditional grading in road construction. The company has embraced new technologies to test autonomous machines with the goals to achieve increased safety, productivity, and uptime. The technologies encompass machine and fleet control systems and logistic solutions for electric equipment with the aim to electrify each transport stage in a quarry – from excavation to primary crushing, and transport to secondary crushing. Moving toward an “emission-free” quarry, Volvo’s successful Electric Site, comprising prototypical electric and autonomous CE machines, new work methods, and management systems, resulted in a 98% reduction in noise and carbon emissions, a 70% reduction in energy cost, and a 40% reduction in operator cost [104].

To achieve high performance of its construction machinery in challenging conditions, Liebherr has developed RAS-based control systems to increase the productivity of excavation works. The optimized and intelligent systems of those excavators increased efficiency (lower fuel consumption and higher load capacity) and improved human ergonomic workspace (optimum visibility with multiple cameras, upper carriage design, cabin’s air condition, and high-resolution touchscreen). The company is continuing to increase technical measures and the constant expansion of the service network, develop hybrid drives for diesel-electric propulsion, as well as focus on automation and digitalization for construction sites.

From a 2017 review on heavy equipment manufacturers’ statistics [105], the remaining companies after the top 10 constituted 39.1% of the global share. A majority of the products have been already available commercial-off-the-shelf (COTS) in the market featuring RAS-based units or control systems to assist the operators. Most of those products, mainly by OEMs mentioned above, are often in line with their strategic development plans. Some smaller companies, such as Autonomous Solutions Inc. (ASI), have added autonomy on existing construction equipment a hardware and software system that can be retrofitted to current vehicles. Their developments have been used in the mine clean-up and other hazardous tasks in the remote control mode. They can also serve in agriculture, industrial cleaning, and military applications. ASI Robotics and Hard-Line Inc. have developed tele-operation kits intended to be universally installed on any earthmoving platform.

To contribute to a cleaner and quieter environment, without compromising safety, productivity, or efficiency, a company specialized in mining and rock excavation, Sandvik Group, has produced electric loaders and RAS-based AutoMine products for the tele-operation or semiautonomous modes of control [65]. The benefit, as claimed, can achieve a (20–50)% decrease in costs per tonne with their digital technologies. Another company for earthmoving automation, Novatron, has developed a touchscreen Display Unit that can incorporate many assist functions, for example, focusing on dealing with a volatile environment.

The application of robotics and automation to earthmoving is expected to disruptively change the mining industry in the next 15 years [106]. Although RAS applied to earthmoving tasks may appear to be technologically common across platforms, the level of automation exhibited by each platform and the modes of control can be different, depending on the market. In this regard, more rooms are given to small- and medium-sized enterprises to develop their own products for the retrofit purpose, for example, functional assist units, offsite control systems, tele-operated modules, sensing and data gathering, AI and cognitive systems, analysis and prediction devices as well as proprietary software tools in order to support earthmoving in volatile environments and increase teaming capability of cooperative machines.

Embracing advances in the ICT and robotics, research, and development directions to be aimed by academia and industry for the sectors involving earthmoving should be to address concerns on environmental sustainability while enhancing productivity, efficiency, and especially safety. To this end, the emerging trend is the electrification of heavy equipment [107], which will not only cut the carbon emissions on a construction or mining site but also offer green solutions to earthmoving problems, facilitated by technologies in ICT robotics and automation [10]. The focus will be on the three technological themes, namely connectivity, electro-mobility, and automation, as identified by Volvo CE [108]. The developments will cover all modes of control, but particularly the cooperation of well-connected platforms along with promote a new generation of battery-powered earthmoving equipment operating at a higher level of autonomy.

6. Conclusion

There have been rapid developments in earthmoving automation for equipment using robotics and automation systems in various modes of control. On the one hand, difficulties have arisen in face of volatility of the work environment, but on the other hand, advances in perception, control, and communications technologies offer opportunities to overcome. Moreover, the capabilities of teaming earthmoving platforms are also being enhanced by application of the emerging networked robotics, IoT, and AI. Development of RAS-based technologies to tackle these key problems has significantly contribute to address priorities in earthmoving, namely safety, efficiency, and productivity.

Reflecting on all these, this survey article has undertaken a comprehensive overview of recent developments reported in the literature or public archives of relevant stakeholders. From a systematic analysis of challenges and opportunities, the paper also presents some industrial perspectives from equipment manufacturers and construction and highlights to look a head at research and development in this area.

Volatile environments due to unpredictable events, extreme weather conditions, or some unexpected hazards are identified as seriously affecting the earthmoving automation processes, for example, collisions of autonomous machines due to heavy rains or a communication failure. Owing to advances in sensing, IoT-enabled communications, and software tools, self-driving vehicles on an earthmoving site can be informed of local climate conditions and made to work more reliably to increase safety in all modes of control.

As mining and construction are important sectors, increasing productivity and enhanced safety, efficiency of earthmoving while ensuring conditions for sustainable development in face of the urbanization trend are among priorities in the economy of each nation. Research and development in earthmoving automation remain to be attractive for which robotics and ICT continue to play a key role.