Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Vision based perception systems for unmanned forestry machines
Luleå University of Technology, Department of Engineering Sciences and Mathematics, Product and Production Development.ORCID iD: 0000-0001-9965-6955
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

After years of continuous development in the past decades, the field of forest production has achieved a notable degree of mechanization. Nonetheless, the unceasing demand for increased productivity in forestry operations continues to drive the rapid expansion of forestry automation. Critical aspects of forest production, including activities such as harvesting, forwarding, seedling planting, and soil preparation, hold significant potential for improved automation. These advancements promise numerous advantages, such as heightened production efficiency, reduced environmental impact, reduced operational costs, and the creation of a more favourable work environment. Beyond fostering industrial progress, such innovations could also make a positive contribution to the sustainable development of forestry.

Forestry operations take place in intricate natural forest environments, rendering the automation of these processes complex and challenging. To realize full automation of forest machines, it is imperative to equip them with environmental awareness and cognitive capabilities. This can be achieved by integrating advanced imaging sensors with sophisticated algorithms, thereby enabling forest machinery to possess a vision system. By utilizing both novel and existing solutions for object detection, positioning, categorization, and status analysis within the machine's surroundings, combined with intelligent decision-making and control mechanisms, forest machinery can attain a higher degree of operational automation.

This thesis primarily focuses on the development, deployment, and validation of vision systems on an unmanned forestry machine platform, with a special emphasis on the prototype development of essential features that are still manually executed in contemporary forestry operations. These features include forest terrain obstacle detection, roundwood pose estimation, automated selection of seedling planting locations, and autonomous obstacle avoidance for mounders.

In this thesis, vision systems rooted in color and stereo camera sensing are designed and deployed on an unmanned forest machinery platform. These systems encompass the following key functionalities:

•             A precise positioning system for detecting forest terrain obstacles, such as stones and stumps, using stereo camera data in conjunction with deep learning techniques.

•             Localization and pose estimation capabilities for ground logs, leveraging stereo cameras and deep learning.

•             An analysis of the planning area for obstacle detection, along with the extraction of feasible planting zones for establishing seedling planting locations.

•             The development of an automated obstacle avoidance system for forestry mounders, powered by visual solutions.

Through rigorous testing in real-world scenarios spanning logging, loading, planting, and site preparation, this paper demonstrates the feasibility and practicality of enhancing the level of automation in forest machinery operations through the integration of vision systems. It culminates in the creation of tangible and efficient functional prototypes, leading towards a new era of automation in the field of forestry.

Place, publisher, year, edition, pages
Luleå: Luleå University of Technology, 2023.
Series
Doctoral thesis / Luleå University of Technology 1 jan 1997 → …, ISSN 1402-1544
National Category
Vehicle Engineering
Research subject
Machine Design
Identifiers
URN: urn:nbn:se:ltu:diva-101829ISBN: 978-91-8048-420-6 (print)ISBN: 978-91-8048-421-3 (electronic)OAI: oai:DiVA.org:ltu-101829DiVA, id: diva2:1807829
Public defence
2024-01-11, E632, Luleå tekniska universitet, Luleå, 09:00 (English)
Opponent
Supervisors
Available from: 2023-10-30 Created: 2023-10-27 Last updated: 2023-12-08Bibliographically approved
List of papers
1. Implementation of a System for Real-Time Detection and Localization of Terrain Objects on Harvested Forest Land
Open this publication in new window or tab >>Implementation of a System for Real-Time Detection and Localization of Terrain Objects on Harvested Forest Land
2021 (English)In: Forests, ISSN 1999-4907, E-ISSN 1999-4907, Vol. 12, no 9, article id 1142Article in journal (Refereed) Published
Abstract [en]

Research highlights: An automatic localization system for ground obstacles on harvested forest land based on existing mature hardware and software architecture has been successfully implemented. In the tested area, 98% of objects were successfully detected and could on average be positioned within 0.33 m from their true position in the full range 1–10 m from the camera sensor. Background and objectives: Forestry operations in forest environments are full of challenges; detection and localization of objects in complex forest terrains often require a lot of patience and energy from operators. Successful automatic real-time detection and localization of terrain objects not only can reduce the difficulty for operators but are essential for the automation of harvesting and logging tasks. We intend to implement a system prototype that can automatically locate ground obstacles on harvested forest land based on accessible hardware and common software infrastructure. Materials and Methods: An automatic object detection and localization system based on stereo camera sensing is described and evaluated in this paper. This demonstrated system detects and locates objects of interest automatically utilizing the YOLO (You Only Look Once) object detection algorithm and derivation of object positions in 3D space. System performance is evaluated by comparing the automatic detection results of the tests to manual labeling and positioning results. Results: Results show high reliability of the system for automatic detection and location of stumps and large stones and shows good potential for practical application. Overall, object detection on test tracks was 98% successful, and positional location errors were on average 0.33 m in the full range from 1–10 m from the camera sensor. Conclusions: The results indicate that object detection and localization can be used for better operator assessment of surroundings, as well as input to control machines and equipment for object avoidance or targeting.

Place, publisher, year, edition, pages
MDPI, 2021
Keywords
automation, object identification, obstacle detection, terrain obstacle
National Category
Forest Science Computer Vision and Robotics (Autonomous Systems)
Research subject
Machine Design
Identifiers
urn:nbn:se:ltu:diva-86809 (URN)10.3390/f12091142 (DOI)000701035100001 ()2-s2.0-85114723919 (Scopus ID)
Funder
Swedish Energy Agency, 48003-1Vinnova, 2020-04202EU, Horizon 2020, H2020-EU.3.5.1.European Regional Development Fund (ERDF), NYPS 20202905
Note

Validerad;2021;Nivå 2;2021-09-01 (johcin)

Available from: 2021-08-25 Created: 2021-08-25 Last updated: 2023-10-27Bibliographically approved
2. Realization of Autonomous Detection, Positioning and Angle Estimation of Harvested Logs
Open this publication in new window or tab >>Realization of Autonomous Detection, Positioning and Angle Estimation of Harvested Logs
2023 (English)In: Croatian Journal of Forest Engineering, ISSN 1845-5719, E-ISSN 1848-9672, Vol. 44, no 2, p. 369-383Article in journal (Refereed) Published
Abstract [en]

To further develop forest production, higher automation of forest operations is required. Such endeavour promotes research on unmanned forest machines. Designing unmanned forest machines that exercise forwarding requires an understanding of positioning and angle estimations of logs after cutting and delimbing have been conducted, as support for subsequent crane loading work. This study aims to improve the automation of the forwarding operation and presents a system to realize real-time automatic detection, positioning, and angle estimation of harvested logs implemented on an existing unmanned forest machine experimental platform from the AORO (Arctic Off-Road Robotics) Lab. This system uses ROS as the underlying software architecture and a Zed2 camera and NVIDIA JETSON AGX XAVIER as the imaging sensor and computing platform, respectively, utilizing the YOLOv3 algorithm for real-time object detection. Moreover, the study combines the processing of depth data and depth to spatial transform to realize the calculation of the relative location of the target log related to the camera. On this basis, the angle estimation of the target log is further realized by image processing and color analysis. Finally, the absolute position and log angles are determined by the spatial coordinate transformation of the relative position data. This system was tested and validated using a pre-trained log detector for birch with a mean average precision (mAP) of 80.51%. Log positioning mean error did not exceed 0.27 m and the angle estimation mean error was less than 3 degrees during the tests. This log pose estimation method could encompass one important part of automated forwarding operations.

Place, publisher, year, edition, pages
University of Zagreb, 2023
Keywords
log detection, autonomous forwarding, log grasping
National Category
Computer Vision and Robotics (Autonomous Systems) Forest Science
Research subject
Machine Design
Identifiers
urn:nbn:se:ltu:diva-89939 (URN)10.5552/crojfe.2023.2056 (DOI)001088744700013 ()2-s2.0-85165462778 (Scopus ID)
Funder
VinnovaSwedish Energy AgencyInterreg
Note

Validerad;2023;Nivå 2;2023-06-27 (hanlid)

Available from: 2022-03-29 Created: 2022-03-29 Last updated: 2024-03-07Bibliographically approved
3. Vision Based Planting Position Selection System for an Unmanned Reforestation Machine
Open this publication in new window or tab >>Vision Based Planting Position Selection System for an Unmanned Reforestation Machine
(English)Manuscript (preprint) (Other academic)
National Category
Forest Science Computer Vision and Robotics (Autonomous Systems)
Research subject
Machine Design
Identifiers
urn:nbn:se:ltu:diva-101678 (URN)
Funder
Interreg, 20357984
Available from: 2023-10-16 Created: 2023-10-16 Last updated: 2023-10-27Bibliographically approved
4. Obstacle detection vision system enabling autonomous mounding on clearcuts
Open this publication in new window or tab >>Obstacle detection vision system enabling autonomous mounding on clearcuts
2023 (English)In: Proceedings of the 16th European-African Regional Conference of the ISTVS, International Society for Terrain-Vehicle Systems , 2023Conference paper, Published paper (Refereed)
Place, publisher, year, edition, pages
International Society for Terrain-Vehicle Systems, 2023
National Category
Computer Vision and Robotics (Autonomous Systems) Other Mechanical Engineering
Research subject
Machine Design
Identifiers
urn:nbn:se:ltu:diva-101677 (URN)10.56884/VQZG2856 (DOI)
Conference
16th European-African Regional Conference of the ISTVS, Lublin, Poland, October 11-13 2023
Funder
Vinnova, 2020-04202Interreg, 20357984
Note

Funder: EU (869580)

Available from: 2023-10-16 Created: 2023-10-16 Last updated: 2023-10-27Bibliographically approved
5. Forest Terrain Object Detection Based on RGB and Depth Information
Open this publication in new window or tab >>Forest Terrain Object Detection Based on RGB and Depth Information
2021 (English)In: Proceedings of the 20th International Conference and 9th Americas Conference of the International Society for Terrain-Vehicle Systems / [ed] Martelli M.; Kovecses J.; Shenvi M.; Dixon J., International Society for Terrain-Vehicle Systems , 2021Conference paper, Published paper (Refereed)
Abstract [en]

At present, there is extensive ongoing research and development on autonomous road vehicles, but much less effort is put on autonomous driving in off-road environments, especially with regard to autonomous machines driving in forest terrain. At the same time, forest machine work is harsh for the operator, and the operator’s maximum performance is a major bottleneck for machine operations. One way to enable a safe work environment is to remove the driver from the machine. Thus, forestry machines must become autonomous in order to improve personnel safety and work efficiency. The basis of enabling automation of forest machinery is highly precise and accurate acquisition of working environment information, especially reliable virtual representation of forest terrain and obstacles within. In this paper, the objective is to improve the accuracy of object detection in forest terrain based on an AI algorithm by adding the depth variation texture to corresponding area of  RGB images. The results show that the algorithm significantly improves the accuracy of object detection while the probability of false detection is reduced. Meanwhile, computational efficiency compared to previous image processing is not reduced to such extent that it prevents real-time computation.

Place, publisher, year, edition, pages
International Society for Terrain-Vehicle Systems, 2021
Keywords
Forest Machine, Stump Detection, Machine Learning, Stereo Camera
National Category
Computer Vision and Robotics (Autonomous Systems)
Research subject
Machine Design
Identifiers
urn:nbn:se:ltu:diva-83698 (URN)2-s2.0-85124530071 (Scopus ID)
Conference
20th International Conference and 9th Americas Conference of the International Society for Terrain-Vehicle Systems (ISTVS 2021), [Online], September 27-29, 2021
Funder
Interreg NordVinnovaEU, Horizon 2020, 869580
Available from: 2021-04-15 Created: 2021-04-15 Last updated: 2023-10-27Bibliographically approved

Open Access in DiVA

fulltext(7387 kB)161 downloads
File information
File name FULLTEXT01.pdfFile size 7387 kBChecksum SHA-512
8ba33a0de24d5c420ee25c4ab040ed14bf6f5d0fc01695a8601251561267a01264ab06ed14bd5f441d95d5fa57f66c04e07021079db9101c808112fcd0e1d7a2
Type fulltextMimetype application/pdf

Authority records

Li, Songyu

Search in DiVA

By author/editor
Li, Songyu
By organisation
Product and Production Development
Vehicle Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 161 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1671 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf