Open Access Open Access  Restricted Access Subscription or Fee Access

Reinforcement Q-Learning for Path Planning of Unmanned Aerial Vehicles (UAVs) in Unknown Environments


(*) Corresponding author


Authors' affiliations


DOI: https://doi.org/10.15866/ireaco.v16i5.24078

Abstract


Path planning for Unmanned Aerial Vehicles in environments with obstacles remains a challenging task. Traditional algorithms, such as A* and Dijkstra, have limitations when dealing with dynamic and changing obstacles, as well as unknown environments. In this paper, a Q-Learning approach for the path planning of UAVs in obstacle-rich and unknown environments is proposed. The impact of alpha, gamma, epsilon, the initial matrix, and reward parameters on the learning process is investigated to achieve safe and cost-effective paths with reduced execution time. The proposed approach is evaluated through simulations by adjusting the alpha, gamma, epsilon, initial matrix, and reward values, and the results demonstrate the effectiveness of the proposed method. The simulation results show that adjusting all the studied parameters can significantly improve the performance of the proposed approach, leading to paths that meet cost and timing objectives while avoiding obstacles.
Copyright © 2023 Praise Worthy Prize - All rights reserved.

Keywords


Unmanned Aerial Vehicles; Path Planning; Unknown Environments; Q-Learning

Full Text:

PDF


References


S. Aggarwal and N. Kumar, Path planning techniques for unmanned aerial vehicles: A review, solutions, and challenges, Comput. Commun. 149, 270-299 (2020).
https://doi.org/10.1016/j.comcom.2019.10.014

Szabo, S., Železník, V., Mako, S., Rabatin, R., Kinematics of Exploration Using Unmanned Aerial Vehicles, (2022) International Review of Aerospace Engineering (IREASE), 15 (5), pp. 244-253.
https://doi.org/10.15866/irease.v15i5.22361

G. G. d. Castro, G. S. Berger, A. Cantieri, M. Teixeira, J. Lima, A. I. Pereira, and M. F. Pinto, Adaptive path planning for fusing rapidly exploring random trees and deep reinforcement learning in an agriculture dynamic environment uavs, Agriculture 13, 354 (2023).
https://doi.org/10.3390/agriculture13020354

C. Yan and X. Xiang, A path planning algorithm for uav based on improved q-learning, in 2018 2nd international conference on robotics and automation sciences (ICRAS), (IEEE, 2018), pp. 1-5.
https://doi.org/10.1109/ICRAS.2018.8443226

M. Shurrab, R. Mizouni, S. Singh, and H. Otrok, Reinforcement learning framework for uav-based target localization applications, Internet Things p. 100867 (2023).
https://doi.org/10.1016/j.iot.2023.100867

Y. Cao and X. Fang, Optimized-weighted-speedy q-learning algorithm for multi-ugv in static environment path planning under anti-collision cooperation mechanism, Mathematics 11, 2476 (2023).
https://doi.org/10.3390/math11112476

A. El Farnane, M. a. Youssefi, A. Mouhsen, and a. ihyaoui, Visual and lidar-based simultaneous localization and mapping for self-driving cars, Int. J. Electr. Comput. Eng. 12, 6284-6292 (2022).
https://doi.org/10.11591/ijece.v12i6.pp6284-6292

R. Alami, H. Hacid, L. Bellone, M. Barcis, and E. Natalizio, Soreo: A system for safe and autonomous drones fleet navigation with reinforcement learning, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37 (2023), pp. 16398-16400.
https://doi.org/10.1609/aaai.v37i13.27058

J. Yang, S. Lu, M. Han, Y. Li, Y. Ma, Z. Lin, and H. Li, Mapless navigation for uavs via reinforcement learning from demonstrations, Sci. China Technol. Sci. pp. 1-8 (2023).
https://doi.org/10.1007/s11431-022-2292-3

A. Souto, R. Alfaia, E. Cardoso, J. Araújo, and C. Francês, UAV path planning optimization strategy: Considerations of urban morphology, microclimate, and energy efficiency using q-learning algorithm, Drones 7, 123 (2023).
https://doi.org/10.3390/drones7020123

D. Zhang, Z. Xuan, Y. Zhang, J. Yao, X. Li, and X. Li, Path planning of unmanned aerial vehicle in complex environments based on state-detection twin delayed deep deterministic policy gradient, Machines 11, 108 (2023).
https://doi.org/10.3390/machines11010108

N. Ali, K. Kamarudin, M. A. A. Bakar, M. H. F. Rahiman, A. Zakaria, S. M. Mamduh, and L. M. Kamarudin, 2D lidar based reinforcement learning for multi-target path planning in unknown environment, IEEE Access (2023).
https://doi.org/10.1109/ACCESS.2023.3265207

M. H. A. Bakar, A. U. bin Shamsudin, R. A. Rahim, Z. A. Soomro, and A. Adrianshah, Comparison method q-learning and sarsa for simulation of drone controller using reinforcement learning, J. Adv. Res. Appl. Sci. Eng. Technol. 30, 69-78 (2023).
https://doi.org/10.37934/araset.30.3.6978

Mlayeh, H., Ghachem, S., Nasri, O., Ben Othman, K., Stabilization of a Quadrotor Vehicle Using PD and Recursive Nonlinear Control Techniques, (2021) International Review of Aerospace Engineering (IREASE), 14 (4), pp. 211-219.
https://doi.org/10.15866/irease.v14i4.19739

Housny, H., Chater, E., El Fadil, H., Feedforward Neural Network Controller for Quadrotor in the Presence of Payload and Wind Disturbances, (2021) International Review of Automatic Control (IREACO), 14 (5), pp. 287-299.
https://doi.org/10.15866/ireaco.v14i5.20480

Kramar, V., Alchakov, V., Kabanov, A., Dudnikov, S., Dmitriev, A., The Design of Optimal Lateral Motion Control of an UAV Using the Linear-Quadratic Optimization Method in the Complex Domain, (2020) International Review of Aerospace Engineering (IREASE), 13 (6), pp. 217-227.
https://doi.org/10.15866/irease.v13i6.19130

Uc Ríos, C., Teruel, P., Use of Unmanned Aerial Vehicles for Calibration of the Precision Approach Path Indicator System, (2021) International Review of Aerospace Engineering (IREASE), 14 (4), pp. 192-200.
https://doi.org/10.15866/irease.v14i4.20709

Keserwani, Z., Saied, M., Francis, C., Medium and Low-Level Energy Saving Control Strategies for Electric-Powered UAVs, (2023) International Review of Automatic Control (IREACO), 16 (2), pp. 54-65.
https://doi.org/10.15866/ireaco.v16i2.23120

Z. Wang, H. Lu, H. Qin, and Y. Sui, Autonomous underwater vehicle path planning method of soft actor-critic based on game training, J. Mar. Sci. Eng. 10, 2018 (2022).
https://doi.org/10.3390/jmse10122018

J. Jiang, X. Zeng, D. Guzzetti, and Y. You, Path planning for asteroid hopping rovers with pre-trained deep reinforcement learning architectures, Acta Astronaut. 171, 265-279 (2020).
https://doi.org/10.1016/j.actaastro.2020.03.007

C. J. Watkins and P. Dayan, Q-learning, Mach. learning 8, 279-292 (1992).
https://doi.org/10.1007/BF00992698

Y. Xu, Y. Wei, K. Jiang, D. Wang, and H. Deng, Multiple uavs path planning based on deep reinforcement learning in communication denial environment, Mathematics 11, 405 (2023).
https://doi.org/10.3390/math11020405

U. Habiba and R. Jahan, Unmanned aerial vehicle (uav)'drones' using machine learning, Available at SSRN 4430575 (2023).
https://doi.org/10.2139/ssrn.4430575

Lamini, C., Benhlima, S., Bekri, M., Q-Free Walk Ant Hybrid Architecture for Mobile Robot Path Planning in Dynamic Environment, (2022) International Journal on Engineering Applications (IREA), 10 (2), pp. 105-115.
https://doi.org/10.15866/irea.v10i2.20443

Aguirre, D., Barón Velandia, J., Salcedo Parra, O., Routing in Elastic Optical Networks Based on Deep Reinforcement Learning for Multi-Agent Systems, (2022) International Review on Modelling and Simulations (IREMOS), 15 (5), pp. 332-339.
https://doi.org/10.15866/iremos.v15i5.22768


Refbacks

  • There are currently no refbacks.


Please send any question about this web site to info@praiseworthyprize.com
Copyright © 2005-2024 Praise Worthy Prize