Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic

Jingzehua Xu; Ziqi Jia; Zekai Zhang; Tianyu Xing; Jingjing Wang; Yong Ren

doi:10.1007/978-3-031-72341-4_20

Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic

Jingzehua Xu
, Ziqi Jia
, Zekai Zhang
, Tianyu Xing
, Jingjing Wang^*
, Yong Ren

^*Corresponding author for this work

School of Cyber Science and Technology

Tsinghua University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Navigating unmanned surface vehicles (USVs) efficiently and robustly in the presence of obstacles and ocean current interference in marine environments is highly challenging. To achieve robust navigation without environment maps and prior information, we follow the three effective improvements of the distributed algorithm distributional soft actor-critic with three refinements (DSACT) over distributional soft actor-critic (DSAC): expected value substitution, double value distribution learning, and variance-based critic gradient adjustment. In order to further optimize the learning rate of DSACT, we optimize DSACT through loss-adjusted prioritized experience replay (LAP) and propose a local path planner called LAP-DSACT for USV navigation. In order to offset the disturbance of ocean currents and plan a smooth and safe trajectory, the motion compensation is considered in the USV motion model. The experimental results clearly demonstrate that LAP-DSACT algorithm outperforms the comparison algorithms in terms of task time, energy efficiency, and the quality of path.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings
Editors	Michael Wand, Jürgen Schmidhuber, Michael Wand, Kristína Malinovská, Jürgen Schmidhuber, Igor V. Tetko, Igor V. Tetko
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	291-305
Number of pages	15
ISBN (Print)	9783031723407
DOIs	https://doi.org/10.1007/978-3-031-72341-4_20
State	Published - 2024
Event	33rd International Conference on Artificial Neural Networks, ICANN 2024 - Lugano, Switzerland Duration: 17 Sep 2024 → 20 Sep 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15019 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	33rd International Conference on Artificial Neural Networks, ICANN 2024
Country/Territory	Switzerland
City	Lugano
Period	17/09/24 → 20/09/24

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 7 Affordable and Clean Energy
SDG 14 Life Below Water

Keywords

Distributional reinforcement learning
Loss-adjusted prioritized experience replay
Robust navigation
Unmanned surface vehicle

Access to Document

10.1007/978-3-031-72341-4_20

Cite this

Xu, J., Jia, Z., Zhang, Z., Xing, T., Wang, J., & Ren, Y. (2024). Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic. In M. Wand, J. Schmidhuber, M. Wand, K. Malinovská, J. Schmidhuber, I. V. Tetko, & I. V. Tetko (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings (pp. 291-305). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15019 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-72341-4_20

Xu, Jingzehua ; Jia, Ziqi ; Zhang, Zekai et al. / Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic. Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. editor / Michael Wand ; Jürgen Schmidhuber ; Michael Wand ; Kristína Malinovská ; Jürgen Schmidhuber ; Igor V. Tetko ; Igor V. Tetko. Springer Science and Business Media Deutschland GmbH, 2024. pp. 291-305 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{f55f55a1179349df9f0e6156099dc06c,

title = "Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic",

abstract = "Navigating unmanned surface vehicles (USVs) efficiently and robustly in the presence of obstacles and ocean current interference in marine environments is highly challenging. To achieve robust navigation without environment maps and prior information, we follow the three effective improvements of the distributed algorithm distributional soft actor-critic with three refinements (DSACT) over distributional soft actor-critic (DSAC): expected value substitution, double value distribution learning, and variance-based critic gradient adjustment. In order to further optimize the learning rate of DSACT, we optimize DSACT through loss-adjusted prioritized experience replay (LAP) and propose a local path planner called LAP-DSACT for USV navigation. In order to offset the disturbance of ocean currents and plan a smooth and safe trajectory, the motion compensation is considered in the USV motion model. The experimental results clearly demonstrate that LAP-DSACT algorithm outperforms the comparison algorithms in terms of task time, energy efficiency, and the quality of path.",

keywords = "Distributional reinforcement learning, Loss-adjusted prioritized experience replay, Robust navigation, Unmanned surface vehicle",

author = "Jingzehua Xu and Ziqi Jia and Zekai Zhang and Tianyu Xing and Jingjing Wang and Yong Ren",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 33rd International Conference on Artificial Neural Networks, ICANN 2024 ; Conference date: 17-09-2024 Through 20-09-2024",

year = "2024",

doi = "10.1007/978-3-031-72341-4\_20",

language = "英语",

isbn = "9783031723407",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "291--305",

editor = "Michael Wand and J{\"u}rgen Schmidhuber and Michael Wand and Krist{\'i}na Malinovsk{\'a} and J{\"u}rgen Schmidhuber and Tetko, \{Igor V.\} and Tetko, \{Igor V.\}",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings",

address = "德国",

}

Xu, J, Jia, Z, Zhang, Z, Xing, T, Wang, J & Ren, Y 2024, Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic. in M Wand, J Schmidhuber, M Wand, K Malinovská, J Schmidhuber, IV Tetko & IV Tetko (eds), Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15019 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 291-305, 33rd International Conference on Artificial Neural Networks, ICANN 2024, Lugano, Switzerland, 17/09/24. https://doi.org/10.1007/978-3-031-72341-4_20

Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic. / Xu, Jingzehua; Jia, Ziqi; Zhang, Zekai et al.
Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. ed. / Michael Wand; Jürgen Schmidhuber; Michael Wand; Kristína Malinovská; Jürgen Schmidhuber; Igor V. Tetko; Igor V. Tetko. Springer Science and Business Media Deutschland GmbH, 2024. p. 291-305 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15019 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic

AU - Xu, Jingzehua

AU - Jia, Ziqi

AU - Zhang, Zekai

AU - Xing, Tianyu

AU - Wang, Jingjing

AU - Ren, Yong

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - Navigating unmanned surface vehicles (USVs) efficiently and robustly in the presence of obstacles and ocean current interference in marine environments is highly challenging. To achieve robust navigation without environment maps and prior information, we follow the three effective improvements of the distributed algorithm distributional soft actor-critic with three refinements (DSACT) over distributional soft actor-critic (DSAC): expected value substitution, double value distribution learning, and variance-based critic gradient adjustment. In order to further optimize the learning rate of DSACT, we optimize DSACT through loss-adjusted prioritized experience replay (LAP) and propose a local path planner called LAP-DSACT for USV navigation. In order to offset the disturbance of ocean currents and plan a smooth and safe trajectory, the motion compensation is considered in the USV motion model. The experimental results clearly demonstrate that LAP-DSACT algorithm outperforms the comparison algorithms in terms of task time, energy efficiency, and the quality of path.

AB - Navigating unmanned surface vehicles (USVs) efficiently and robustly in the presence of obstacles and ocean current interference in marine environments is highly challenging. To achieve robust navigation without environment maps and prior information, we follow the three effective improvements of the distributed algorithm distributional soft actor-critic with three refinements (DSACT) over distributional soft actor-critic (DSAC): expected value substitution, double value distribution learning, and variance-based critic gradient adjustment. In order to further optimize the learning rate of DSACT, we optimize DSACT through loss-adjusted prioritized experience replay (LAP) and propose a local path planner called LAP-DSACT for USV navigation. In order to offset the disturbance of ocean currents and plan a smooth and safe trajectory, the motion compensation is considered in the USV motion model. The experimental results clearly demonstrate that LAP-DSACT algorithm outperforms the comparison algorithms in terms of task time, energy efficiency, and the quality of path.

KW - Distributional reinforcement learning

KW - Loss-adjusted prioritized experience replay

KW - Robust navigation

KW - Unmanned surface vehicle

UR - https://www.scopus.com/pages/publications/85205316079

U2 - 10.1007/978-3-031-72341-4_20

DO - 10.1007/978-3-031-72341-4_20

M3 - 会议稿件

AN - SCOPUS:85205316079

SN - 9783031723407

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 291

EP - 305

BT - Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings

A2 - Wand, Michael

A2 - Schmidhuber, Jürgen

A2 - Wand, Michael

A2 - Malinovská, Kristína

A2 - Schmidhuber, Jürgen

A2 - Tetko, Igor V.

PB - Springer Science and Business Media Deutschland GmbH

T2 - 33rd International Conference on Artificial Neural Networks, ICANN 2024

Y2 - 17 September 2024 through 20 September 2024

ER -

Xu J, Jia Z, Zhang Z, Xing T, Wang J, Ren Y. Robust Navigation for Unmanned Surface Vehicle Utilizing Improved Distributional Soft Actor-Critic. In Wand M, Schmidhuber J, Wand M, Malinovská K, Schmidhuber J, Tetko IV, Tetko IV, editors, Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 291-305. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-72341-4_20