Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Ensuring Robot-Human Safety for the BD Spot Using Active Visual Tracking and NMPC With Velocity Obstacles
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Signals and Systems.ORCID iD: 0000-0002-1046-0305
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Signals and Systems.ORCID iD: 0000-0003-3922-1735
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Signals and Systems.ORCID iD: 0000-0003-0126-1897
2022 (English)In: IEEE Access, E-ISSN 2169-3536, Vol. 10, p. 100224-100233Article in journal (Refereed) Published
Abstract [en]

When humans and robots operate in and occupy the same local space, proximity detection and proactive collision avoidance is of high importance. As legged robots, such as the Boston Dynamics (BD) Spot, start to appear in real-world application environments, ensuring safe robot-human interactions while operating in full autonomy mode becomes a critical gate-keeping technology for trust in robotic workers. Towards that problem, this article proposes a track-and-avoid architecture for legged robots that combines a visual object detection and estimation pipeline with a Nonlinear Model Predictive Controller (NMPC) based on the Optimization Engine, capable of generating trajectories that satisfy the avoidance and tracking problems in real-time operations where the computation time never exceeded 40 ms. The system is experimentally evaluated using the BD Spot, in a custom sensor and computational suite, and in fully autonomous operational conditions, for the robot-human safety scenario of quickly moving noncooperative obstacles. The results demonstrate the efficacy of the scheme in multiple scenarios where the maximum safety distance violation was only 9 cm for an obstacle moving at 2.5 m/s while affected by both state estimation and object detection uncertainty and noise.

Place, publisher, year, edition, pages
IEEE, 2022. Vol. 10, p. 100224-100233
Keywords [en]
Human robot interaction, NMPC, object detection, object tracking, spot, velocity obstacle
National Category
Computer Sciences Control Engineering
Research subject
Robotics and Artificial Intelligence
Identifiers
URN: urn:nbn:se:ltu:diva-93404DOI: 10.1109/access.2022.3205611ISI: 000861351500001Scopus ID: 2-s2.0-85139414977OAI: oai:DiVA.org:ltu-93404DiVA, id: diva2:1701221
Funder
EU, Horizon 2020, 869379EU, Horizon 2020, 101003591
Note

Validerad;2022;Nivå 2;2022-10-05 (joosat);

Available from: 2022-10-05 Created: 2022-10-05 Last updated: 2025-12-10Bibliographically approved
In thesis
1. Risk Aware Path Planning and Dynamic Obstacle Avoidance towards Enabling Safe Robotic Missions
Open this publication in new window or tab >>Risk Aware Path Planning and Dynamic Obstacle Avoidance towards Enabling Safe Robotic Missions
2023 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

This compilation thesis presents two main contributions in path planning and obstacle avoidance, as well as an integration of the proposed modules with other frameworks to enable resilient robotic missions in complex environments.In general, through different types of robotic missions it is important to have a collision tolerant and reliable system, both regarding potential risks from collisions with dynamic and static obstacles, but also to secure the overall mission success.%Particularly, a common trend in the presented work is safety regarding collisions with dynamic and static obstacles, as well as reliable overall systems that are capable of executing missions.

The work included in this thesis presents the risk-aware path planner D$^*_+$ that is capable of planning traversable paths for both ground and aerial robots. D$^*_+$ is developed on top of D$^*$-lite with a risk layer close to occupied space, modeling the unknown areas as a risk, and is implemented with a dynamic map to enable updates and adjustments to a changing environment.

The risk layer aids in solving two common challenges with path planning for real robots: a) it creates a safety margin that gives free space between the path and obstacles so that robots with the corresponding size can follow the path, and b) it masks smaller holes in walls that occur when building maps from real data.

Using a dynamic map makes it possible to use D$^*_+$ for an exploration mission, it also enables for the re-planning of the path if the environment changes for example, if an obstacle suddenly blocks a path, a new path will be planned. D$^*_+$ have been tested in different real-life experiments with both an Unmanned Areal Vehicle (UAV) and a quadruped-legged robot and shown to produce traversable paths in different application scenarios, such as exploration, return to base, and navigation on known maps.

This thesis also presents an obstacle avoidance architecture for velocity objects, structured around an object detection and tracking scheme that is combined with non-linear model predictive controller (NMPC) to plan the avoidance maneuver. %that uses a Convolutional Neural Network to detect obstacles that are tracked so they can be avoided by a non-linear model predictive controller (NMPC).In this case, the detection is done with the Convoluitonal Neural Network (CNN) You Only Lock Once v4 (YOLO) where the most certain human is tracked with a Kalman filter, and the velocity of the human is estimated.The proposed scheme models the object motion as constant velocity, which is utilized from the NMPC to plan control inputs for the robot to avoid the identified obstacle. A merit of the approach is that the avoidance maneuver does not only consider the current identification position, but also considers the motion prediction of the object. This avoidance framework proved to be capable to avoid non-cooperative obstacles, such as humans moving towards it.Due to the fact that the avoidance is starting when a future collision is predicted, the avoidance maneuver is started early enough to avoid obstacles with a higher velocity than a classic ``static obstacle'' radius approach can handle.

An additional aim of this thesis is to showcase that the proposed contributions can be applied in full robotic missions/frameworks. Thus, this thesis presents a search and rescue mission with a quadruped-legged robot and a UAV on a partially known map to find the location of survivors and other objects of related interest. In this mission, the quadruped-legged robot carries the UAV to the edge of the known map from where it launches the UAV that then explores and detects any survival and other relevant objects.Also, an autonomy solution, based on Boston dynamics' quadruped-legged robot Spot, for enabling a map-based navigation in confined environments has been developed and tested. This Spot solution enables the robot to navigate to a user-selected point, rotate in the desired direction, and instruct the UAV, in the combined search and rescue mission, to take off.

Place, publisher, year, edition, pages
Luleå University of Technology, 2023
Series
Licentiate thesis / Luleå University of Technology, ISSN 1402-1757
Keywords
Robotic, Path planning, Obstacle avoidanc, Robotic missions, licenti thesis
National Category
Robotics and automation
Research subject
Robotics and Artificial Intelligence
Identifiers
urn:nbn:se:ltu:diva-95349 (URN)978-91-8048-248-6 (ISBN)978-91-8048-249-3 (ISBN)
Presentation
2023-03-09, A1545, Luleå tekniska universitet, Luleå, 09:00 (English)
Opponent
Supervisors
Available from: 2023-01-24 Created: 2023-01-20 Last updated: 2025-10-21Bibliographically approved
2. Safe and Field Resilient Risk-Aware Path Planning with Dynamic Obstacle Avoidance in Unknown and Uncontrolled Environments
Open this publication in new window or tab >>Safe and Field Resilient Risk-Aware Path Planning with Dynamic Obstacle Avoidance in Unknown and Uncontrolled Environments
2026 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

This PhD thesis advances robotic autonomy by developing novel path-planning and collision-avoidance solutions that enable resilient missions in complex, unstructured real-world environments. The primary contribution is D*+, a risk-aware path planner extending the D*-lite framework for ground and aerial robots. D*+ introduces a risk layer around occupied and unknown spaces, ensuring traversable paths with safety margins while operating on imperfect maps from real data. Its dynamic mapping supports adaptive replanning, enabling exploration missions in unknown environments, and is excellent for waypoint navigation. Real-world trials with a UAV and quadrupedal robot confirm its versatility across diverse scenarios.

The second contribution is the Detect Track and Avoid Architecture (DTAA), which tackles dynamic obstacles using YOLO-based detection, Kalman filter state estimation, and a nonlinear model predictive controller (NMPC) for anticipatory avoidance maneuvers. DTAA effectively handles fast-moving objects while following D*+ paths; however, it is limited by a short predictive horizon and susceptible to local minima. To overcome these weaknesses, this thesis introduces A*+T, a distributed, time-dependent multi-agent path planner. Built on an A*framework, A*+T integrates D*+ 's risk layers and DTAA's dynamic obstacle handling, adding a temporal dimension to the planning process, enabling collision checks in time and space. The temporal dimension enables distributed autonomous robots to plan collision-free paths in shared spaces based on other robots' planned paths.

Leveraging shared paths and predicted paths from DTAA, A*+T plans collision-free paths around the dynamic obstacles. Validated through simulations and real-world experiments, A*+T enhances mission readiness for multi-agent scenarios.

Beyond these, the thesis integrates these modules into complete robotic systems, enhancing mission control for large-scale applications. Demonstrations include mining inspections (visual and gas detection) and search-and-rescue missions (locating humans/objects). These original advancements offer robust, practical solutions for robotic navigation, validated through extensive real-world testing, and contribute significantly to autonomous systems in high-stakes environments.

Place, publisher, year, edition, pages
Luleå University of Technology, 2026
Series
Doctoral thesis / Luleå University of Technology 1 jan 1997 → …, ISSN 1402-1544
Keywords
Path-planning, Dynamic obstacle avoidance, Field robotics
National Category
Robotics and automation
Research subject
Robotics and Artificial Intelligence
Identifiers
urn:nbn:se:ltu:diva-115766 (URN)978-91-8048-964-5 (ISBN)978-91-8048-965-2 (ISBN)
Public defence
2026-04-14, A1545, Luleå University of Technology, Luleå, 08:30 (English)
Opponent
Supervisors
Available from: 2025-12-10 Created: 2025-12-10 Last updated: 2026-03-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Karlsson, SamuelLindqvist, BjörnNikolakopoulos, George

Search in DiVA

By author/editor
Karlsson, SamuelLindqvist, BjörnNikolakopoulos, George
By organisation
Signals and Systems
In the same journal
IEEE Access
Computer SciencesControl Engineering

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 168 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf