Machine vision is required by autonomous heavy construction equipment to navigate and interact with the environment. Wheel loaders need the ability to identify different objects and other equipment to perform the task of automatically loading and dumping material on dump trucks, which can be achieved using deep neural networks. Training such networks from scratch requires the iterative collection of potentially large amounts of video data, which is challenging at construction sites because of the complexity of safely operating heavy equipment in realistic environments. Transfer learning, for which pretrained neural networks can be retrained for use at construction sites, is thus attractive, especially if data can be acquired without full-scale experiments. We investigate the possibility of using scalemodel data for training and validating two different pretrained networks and use real-world test data to examine their generalization capability. A dataset containing 268 images of a 1:16 scale model of a Volvo A60H dump truck is provided, as well as 64 test images of a full-size Volvo A25G dump truck. The code and dataset are publicly available 1 . The networks, both pretrained on the MS-COCO dataset, were fine-tuned to the created dataset, and the results indicate that both networks can learn the features of the scale-model dump truck (validation mAP of 0.82 for YOLOv3 and 0.95 for RetinaNet). Both networks can transfer these learned features to detect objects on a full-size dump truck with no additional training (test mAP of 0.70 for YOLOv3 and 0.79 for RetinaNet).

Earth-moving machines are heavy-duty vehicles designed for construction operations involving earthworks. The tasks performed by such machines typically involve navigation and interaction with materials such as soil, gravel, and blasted rock. Skilled operators use a combination of visual, sound, tactile and possibly motion feedback to perform tasks efficiently. We survey the literature in this research area and analyse the relative importance of different sensor system modalities focusing on deep-learning-based vision and automation for the short-cycle loading task. This is a common and repetitive task that is attractive to automate. The analysis indicates that computer vision, in combination with onboard sensors, is more critical than coordinate-based positioning. Furthermore, we find that data-driven approaches, in general, have high potential in terms of productivity, adaptability, versatility and wear and tear with respect to automation system solutions. The main knowledge gaps identified relate to loading non-fine heterogeneous material and navigation during loading and unloading.

Data collection and annotation is a time consuming and costly process, yet necessary for machine vision. Automation of construction equipment relies on seeing and detecting different objects in the vehicle’s surroundings. Construction equipment is commonly used to perform frequent repetitive tasks, which are interesting to automate. An example of such a task is the short-loading cycle, where the material is moved from a pile into the tipping body of a dump truck for transport. To complete this task, the wheel loader needs to have the capability to locate the tipping body of the dump truck. The machine vision system also allows the vehicle to detect unforeseen dangers such as other vehicles and more importantly human workers. In this work, we investigate the viability to perform semi-automatic annotation of video data using linear interpolation. The data is collected using scale-models mimicking a wheel-loaders approach towards a dump truck during the short-loading cycle. To measure the viability of this type of solution, the workload is compared to the accuracy of the model, YOLOv3. The results indicate that it is possible to maintain the performance while decreasing the annotation workload by about 95%. This is an interesting result for this application domain, as safety is critical and retaining the vision system performance is more important than decreasing the annotation workload. The fact that the performance seems to retain with a large workload decrease is an encouraging sign.

The short-loading cycle is a construction task where a wheel loader scoops material from a nearby pile in order to move that material to the tipping body of a dump truck. The short-loading cycle is a vital task performed in high quantities and is often part of a more extensive never-ending process to move material for further refinement. This, together with the highly repetitive nature of the short-loading cycle, makes it a suitable candidate for automation. However, the short-loading cycle is a complex task where the mechanics of the wheel loader together with the interaction between the wheel loader and the environment needs to be considered. This must be achieved while maintaining some productivity goal and, concurrently, minimizing the used energy. The main objective of this work is to analyze the short-loading cycle, assess the current state of research in this field, and discuss the steps required to progress towards a minimal viable product consisting of individual automation solutions that can perform the short-loading cycle well enough to be used by early adopters. This is achieved through a comprehensive literature study and consequent analysis of the review results. From this analysis, the requirements of an MVP are defined and some gaps which are currently hindering the realization of the MVP are presented.

The short-loading cycle is a repetitive task performed in high quantities, making it a great alternative for automation. In the short-loading cycle, an expert operator navigates towards a pile, fills the bucket with material, navigates to a dump truck, and dumps the material into the tipping body. The operator has to balance the productivity goal while minimising the fuel usage, to maximise the overall efficiency of the cycle. In addition, difficult interactions, such as the tyre-to-surface interaction further complicate the cycle. These types of hard-to-model interactions that can be difficult to address with rule-based systems, together with the efficiency requirements, motivate us to examine the potential of data-driven approaches. In this paper, the possibility of teaching an agent through reinforcement learning to approach a dump truck's tipping body and get in position to dump material in the tipping body is examined. The agent is trained in a 3D simulated environment to perform a simplified navigation task. The trained agent is directly transferred to a real vehicle, to perform the same task, with no additional training. The results indicate that the agent can successfully learn to navigate towards the dump truck with a limited amount of control signals in simulation and when transferred to a real vehicle, exhibits the correct behaviour.