The use of physical vehicles and environments during vehicle research and development is highly resource-intensive, particularly for autonomous vehicles. Recently, digital models are therefore increasingly used instead, which require high levels of fidelity and validity. While the two aforementioned qualities are often lacking, an absence of versatility for multi-purpose use is even more prevalent in current digital models. In response to these challenges, this work presents a novel real-time multi-physics digital twin of an offroad vehicle with high levels of fidelity and validity, both regarding the vehicle dynamics and hydraulics, as well as regarding the visual representation of the environment and the exteroceptive sensor emulation. The versatility of the digital twin enables its usage for vehicle development tasks concerning mechanical components and driveline, as well as for visual machine learning tasks, such as generation of auto-annotated visual training data. Development of control algorithms leveraging both visual input and mechanical systems is also enabled. Furthermore, the real-time capability allows for Hardware-in-the-Loop and Vehicle-in-the-Loop simulation. The modeling, calibration, and real-world validation of the digital twin is presented, with an emphasis on the vehicle dynamics and hydraulics. The shown validity enables advancements in the development of autonomous offroad vehicles.

Object detectors for autonomous forestry operations have previously been developed mainly by training on physical manually annotated data, which is both time-consuming and costly. Since the ground truth in the virtual model is known, the training data can be auto-annotated, enabling the creation of larger training datasets, while also improving time and cost efficiency. In this work, a virtual environment in Unity is used in co-simulation with a real-time digital twin of a physical forestry vehicle, to generate realistic auto-annotated training data, as captured by an onboard stereo camera. First, it is shown that a log detector trained on physical data can detect logs in the virtual environment. Second, new detectors are trained, using different shares of virtual and physical data. It is shown that a detector trained using only virtual data, can learn to detect logs in the physical world. Moreover, virtual pre-training is shown to improve the performance of physically trained and tested detectors, both at low availability of physical training data, and in terms of domain generalization. A detailed detector performance analysis also highlights further potential and opportunities for future improvements. Furthermore, the real-time capable virtual models enable future machine learning tasks utilizing different levels of Hardware-in-the-Loop. 

An integral part of autonomous forestry is the ability of the vehicles, e.g., forwarders and harvesters, to perceive their environment. At Luleå University of Technology, object detectors have previously been developed, allowing forestry vehicles to detect and position important objects in forestry, such as tree stumps, stones, and logs. These detectors have been developed by training on physical manually annotated data, which is both time-consuming and costly. Training on virtual data allows for significant time- and cost reductions. Since the ground truth in the virtual model is known, the training data can be auto-annotated, allowing for the creation of larger training datasets, at a lower cost. In this work, a virtual environment in Unity is used in co-simulation with a real-time digital twin of a physical forestry vehicle, to generate auto-annotated training data, as captured by an onboard stereo camera. A detailed emulation of the stereo camera is used to achieve realistic results. First, a log detector trained on physical manually annotated data, is evaluated on virtually created data. It is shown that the log detector trained on physical data can detect logs in the virtual environment. Second, new detectors are trained, using different shares of physical and virtual data. It is shown that a detector trained using only virtual data, can learn to detect logs in the physical world. Moreover, virtual pre-training is shown to improve the performance of physically trained and tested detectors, both at low availability of physical training data, and in terms of domain generalization. Furthermore, the real-time capable virtual models also enable future machine learning tasks utilizing different levels of Hardware-in-the-Loop.

Abstract: At Luleå University of Technology, an offroad vehicle capable of performing autonomous forwarding of full-scale logs, has been developed and built. This vehicle serves as a modular research platform, where different hardware, software, and algorithms can be exchanged, tested, and validated. To further facilitate this research, a real-time multi-physics digital twin of the vehicle has been developed. The digital twin can be executed either in fully virtual or in Hardware-in-the-Loop mode. Virtual and hybrid testing and development are expected to increase the efficiency, safety, and sustainability of these activities. Future envisioned use-cases also include machine learning and generation of auto-annotated detector training data. The digital twin features multibody simulation, hydraulics, contacts and collisions, flexible tires, emulated sensors, and co-simulation capabilities. The test environment is a physical environment that has been recreated digitally, using data from a UAV-mounted camera and lidar, as well as handheld RTK GPS measurements. Therefore, accurate GPS positions can be obtained from the digital environment. The digital twin is equipped with an emulated GPS sensor and can thus follow the same GPS tracks as the physical vehicle. This paper presents the development and verification of the digital twin model. Calibration and testing of the digital twin’s hydraulically actuated articulated joint is detailed, and the results are analyzed. The modeled hydraulics of the articulated joint show realistic dynamics. It is also shown that the level of detail of the wheel motors’ hydraulic circuits, affects the calibration of the hydraulics model of the articulated joint, as they affect the resistance during turning.