Earth-moving machines are machines used in a wide range of industries, such as the construction industry, to perform tasks related to earthworks.Currently, the vast majority of earth-moving machines are human-operated where expert operators perform these industry vital tasks.One such task is the short-loading cycle which is a repetitive work cycle performed in high quantities within the construction industry.This work cycle aims to use a wheel-loader to move material from a pile or from the ground to the tipping body of a dump truck.Not only is this task repetitive and performed in high quantities, but it is also representative of the knowledge required to perform a wide set of other work cycles, hence a good candidate for automation.

Skilled operators use their sensory input to perform the tasks required, such as tactile, sound and sight.One of the most important senses leveraged during normal operations is sight, as it is used to locate dynamic objects and detect dangers.Thus to be able to replace the driver of an earth-moving machine with an autonomous system, the system requires similar vision capabilities.Machine Vision is a field where the goal is to use some type of vision sensor, such as cameras, to extract relevant high-level information from images or video streams.This thesis aims to examine how machine vision can be used within the short-loading cycle to facilitate performing said work cycle autonomously.

The main findings in this thesis are threefold: Firstly, two knowledge gaps are identified in the domain of automation during the short-loading cycle.These relate to the loading of heterogeneous material and navigation during loading and unloading.Secondly, we show that it is possible to train a deep learning model to detect the cab, wheels and tipping body of a scale-model dump truck while mimicking the approach towards the load carrier during the short-loading cycle.This model can then be applied to real vehicles to detect the same objects, with no additional training.Lastly, we show that linear interpolation can be used to perform semi-automatic labelling of camera-based video data of the approach of a wheel-loader towards a dump truck during the short-loading cycle.This technique decreases the annotation workload by around 95% while retaining comparable performance.

The future direction of this work includes using techniques such as reinforcement learning to teach a model to perform the navigation required during the short-loading cycle.Future work also includes using world models to learn representations of underlying structures in the environment, open-ended learning to transfer the learned knowledge to adjacent work cycles and using machine vision to find the point of attack for scooping heterogeneous material.