Virtual environments for modeling the interaction of operators with UAVs in closed spaces in potentially dangerous situations

Virtual environments for modeling the interaction of operators with UAVs in closed spaces in potentially dangerous situations

Sergey F. Sergeev
Doctor of Psychological Science, Professor, Peter the Great Saint Petersburg Polytechnical University (SPbPU), Head of Laboratory, 29, Politekhnicheskaya ul., Saint Petersburg, 195251, Russia, tel.: +7(911)995-09-29, This email address is being protected from spambots. You need JavaScript enabled to view it., This email address is being protected from spambots. You need JavaScript enabled to view it., ORCID: 0000-0002-6677-8320

Yuri A. Bubeev
Doctor of Medical Science, Professor, Institute of Biomedical Problems (IBMP), Deputy Director for Science, building 4, 76-A, Khoroshevskoe shosse, Moscow, 123007, Russia, tel.: +7(499)195-03-73, This email address is being protected from spambots. You need JavaScript enabled to view it.

Vitaly M. Usov
Doctor of Medical Science, Professor, IBMP, Leading Research Scientist, building 4, 76-A, Khoroshevskoe shosse, Moscow, 123007, Russia, tel.: +7(910)495-48-02, This email address is being protected from spambots. You need JavaScript enabled to view it., ORCID: 0000-0001-9759-3444
Mikhail V. Mikhaylyuk, Doctor of Physical and Mathematical Sciences, Professor, Scientific Research Institute for System Analysis of the Russian Academy of Sciences, Head of Department, 36-1, Nakhimovsky pr., Moscow, 117218, Russia, tel.: +7(499)129-28-30, This email address is being protected from spambots. You need JavaScript enabled to view it., ORCID: 0000-0002-7793-080X

Maxim M. Knyazkov
PhD in Technical Sciences, Ishlinsky Institute for Problems in Mechanics of the Russian Academy of Sciences (IPMech RAS), Senior Research Scientist, 101-1, pr. Vernadskogo, Moscow, 119526, Russia, tel.: +7(495)495-434-77-66, This email address is being protected from spambots. You need JavaScript enabled to view it., ORCID: 0000-0002-5863-4079

Alexey V. Polyakov
PhD in Medical Sciences, IBMP, Head of Department, building 4, 76-A, Khoroshevskoe shosse, Moscow, 123007, Russia, tel.: +7(916)131-80-85, This email address is being protected from spambots. You need JavaScript enabled to view it.

Anna I. Motienko
PhD in Technical Sciences, Saint Petersburg Federal Research Center of the Russian Acad-emy of Sciences, Senior Research Scientist, 39, 14 line V.O., Saint Petersburg, 199178, Russia, This email address is being protected from spambots. You need JavaScript enabled to view it., ORCID: 0000-0002-0315-9485

Alexandr V. Khomyakov
PhD in Technical Sciences, JSC «Central Design Bureau of Apparatus Engineering», General Director, 36, ul. Demonstracii, Tula, 300034, Russia, tel.: +7(4872)55-40-90, This email address is being protected from spambots. You need JavaScript enabled to view it.

Received January 24, 2022.

Abstract
The effective interaction between humans and robotic devices is designed to provide a timely response to potentially dangerous situations arising in complex environments. One example is the use of Unmanned Aerial Vehicles (UAVs) by human operators to inspect enclosed spaces. These units allow the search and identification of target objects, helping a human operator identify signs of Potentially Dangerous Situations (PDSs). At the same time in a closed (and often cluttered) space to control the emerging situations it is necessary to involve developed means of conducting spatial orientation based on Artificial Intelligence Technology, Computer Vision, 3D-visualization, Local Positioning And Navigation, etc. Despite the progress made in autonomous UAVs control, active human participation in detecting signs of PDSs is an important condition for maintaining the security of the internal environment of controlled premises. Human decision-making in situations of high uncertainty and incomplete data largely depends on means to improve his Situational Awareness (external conditions of activity) and Professional Competencies (internal conditions of activity). To form them, it is necessary to develop methodology and tools for building UAVs application scenarios to give operators the necessary user experience on simulation models in the Virtual Environment of Activity. The paper investigated the development of the Virtual Environment Systems (VES) for the implementation of UAVs application scenarios in enclosed spaces in relation to multicopters that allow both autonomous and manual control. As a result of this research, prototypes of UAVs for inspection of enclosed spaces have been identified; the prospects of quadcopters use under various PDSs scenarios have been shown; the composition of promising technologies ensuring UAVs use in enclosed environments has been described; the method of design and 3D visualization of UAVs application scenarios using the original VES (named VirSim) has been proposed. The advantages of the proposed approach to modeling the use of UAVs in PDSs are the multivariate scenarios of operators' activities, which contributes to the accumulation of user experience of human interaction with flying robots.

Key words
Enclosed spaces, potentially dangerous situations, unmanned aerial vehicles, multicopters, automatic and manual control, virtual environments systems, activity scenarios.

DOI
10.31776/RTCJ.10201

Bibliographic description
Sergeev, S. et al., 2022. Virtual environments for modeling the interaction of operators with UAVs in closed spaces in potentially dangerous situations. Robotics and Technical Cybernetics, 10(2), pp.85-92.

UDC identifier:
519.711.3:004.946:623.746.4-519:614.8

References 

  1. Sergeev, S.F., 2009. Obuchayushchie i Professional'nye Immersivnye Sredy [Educational and Professional Immersive Environments]. Moscow: Narodnoe obrazovanie Publ., p.432. (in Russian).
  2. Sergeev, S.F., 2018. Helovecheskij faktor v samoletah 6-go pokoleniya: problemy tekhnosimbioza [The human factor in the aircraft of the 6th generation: problems technosymbiosis]. Mekhatronika, Avtomatizatsiya, Upravlenie [Mechatronics, Automation, Control], 19(12), pp.806-811. DOI: 10.17587/mau.19.806-811. (in Russian).
  3. Sergeev, S.F. et al., 2020. Virtual'nye sredy obucheniya v zadachah osvoeniya Luny s pomoshch'yu robotov [Virtual learning environments in the tasks of lunar exploration using robots]. Robototekhnika i Tekhnicheskaya Kibernetika [Robotics and Technical Cybernetics], 8(3), pp.165-174. DOI: 10.31776/RTCJ.8301. (in Russian).
  4. Polyakov, A.V. et al., 2019. Komp'yuternoe modelirovanie zhizneugrozhayushchih situacij i provedeniya avarijno-spasatel'nyh, medicinskih i evakuacionnyh meropriyatij na lunnoj baze [Computer simulation of life-threatening situations and rescue, medical and evacuation activities on a lunar base]. Aviakosmicheskaya i Ekologicheskaya Meditsina [Aerospace and Environmental Medicine], 53(3), pp.13-19. DOI: 10.21687/0233-528X-2019-53-3-13-19. (in Russian).
  5. Kim, M.L. et al., 2018. O vozmozhnosti ispol'zovaniya robototekhnicheskih letatel'nyh apparatov pri vypolnenii operativnogo plana likvidacii avarii na shahtah [On the possibility of using robotic aircraft in the implementation of the operational plan for the elimination of an accident in mines]. Ugol' [Coal], 1, pp.34–38. (in Russian).
  6. Kim, M.L. et al., 2019. Sistema avtomaticheskogo upravleniya traektornym dvizheniem shahtnogo bespilotnogo letatel'nogo apparata [Automatic trajectory motion control system for mine unmanned aircrafts]. Gornaya promyshlennost' [Mining], 3(145), pp.60-64. DOI: 10.30686/1609-9192-2019-3-145-60-64. (in Russian).
  7. Adamov, A.P., 2017. Analiz ekspluatacii mul'tikopterov s pozicii nadezhnosti i bezopasnosti [Analysis of the operation of multicopters from the standpoint of reliability and safety]. Nadezhnost' i Kachestvo Slozhnykh Sistem [Reliability and Quality of Complex Systems], 3(19), pp.86–93. (in Russian).
  8. Arkhipov, O. et al., 2018. Algoritm obsledovaniya zamknutogo pomeshcheniya avtonomnym mobil'nym robotom [Enclosed room exploration algorithm for an autonomous mobile robot]. Sistemy i Sredstva Informatiki [Systems and Means of Informatics], 28(4), pp.10–21. DOI: 10.14357/08696527180402. (in Russian).
  9. Fong, T. et al., 2013. Smart SPHERES: a telerobotic free-flyer for intravehicular activities in space. In: AIAA Space 2013 Conference and Exposition, pp.5338–5352. DOI: 10.2514/6.2013-5338.
  10. Smith, T. et al., 2016. Astrobee: A new platform for free-flying robotics research on the international space station. In: Proc. Int. Symp. on AI, Robotics, and Automation in Space (iSAIRAS). Available at: <https://ti.arc.nasa.gov/publications/31584/download/> (Accessed 01 June 2022).
  11. Bualat, M., et al., 2018. Astrobee: A new tool for ISS operations. In: Proc. SpaceOps (AIAA 2018-2517). Available at: <https://www.nasa.gov/sites/default/files/atoms/files/bualat_spaceops_2018_paper.pdf> (Accessed 01 June 2022).
  12. Szafir, D., et al., 2017. Designing planning and control interfaces to support user collaboration with flying robots. The International Journal of Robotics Research, 36, pp.514–542. DOI: 10.1177/0278364916688256.
  13. Carlino, R., 2019. Astrobee free-flyers: integrated and tested. ready for launch! In: IAF Space Systems Symposium. Cooperative and Robotic Space Systems. Available at: <https://iafastro.directory/iac/paper/id/52291/abstract-pdf/IAC-19,D1,6,4,x52291.brief.pdf> (Accessed 01 June 2022).
  14. Fluckiger, L. and Coltin, B., 2019. Astrobee Robot Software: Enabling Mobile Autonomy on the ISS. Available at: <https://www.nasa.gov/sites/default/files/atoms/files/fluckiger2018astrobee.pdf> (Accessed 01 June 2022).
  15. Serrano, J. et al., 2020. Cognitive architectural control for free-flying robots on the lunar orbital platform-gateway. In: Proceedings: 12th International Conference, ICSR 2020. DOI:10.1007/978-3-030-62056-1.
  16. Bubeev, Yu.A. et al., 2020. Sposobnost' operatora k vedeniyu prostranstvennoj orientirovki pri distancionnom upravlenii bespilotnymi apparatami v virtual'noj srede [The ability of the operator to maintain spatial orientation during remote control of unmanned aerial vehicles in a virtual environment]. Aviakosmicheskaya i Ekologicheskaya Meditsina [Aerospace and Environmental Medicine], 55(3), pp.16-27. DOI: 10.21687 / 0233-528X-2021-55-3-16-27. (in Russian).
  17. Mikhayluk, M.V. et al., 2020. Sistema virtual'nogo okruzheniya VirSim dlya imitacionno-trenazhernyh kompleksov podgotovki kosmonavtov [VirSim virtual environment system for simulated cosmonaut training complexes]. Pilotiruemye Polety v Kosmos [Manned Spaceflights], 4(37), pp.72-95. DOI: 10.34131/MSF.20.4.72-95. (in Russian).
  18. Bouabdallah, S., 2007. Design and Control of Quadrotors with Application to Autonomous Flying. MSс, Dept. Autonomous system Eng., Aboubekr Belkaid Univ. Tlemcen, Algérie Université Belkaid. DOI:10.5075/epfl-thesis-3727.
  19. Nieuwenhuisen, M., et al., 2014. Obstacle detection and navigation planning for autonomous micro aerial vehicles. In: 2014 International Conference on Unmanned Aircraft Systems, pp.1040–1047. DOI: 10.1109/ICUAS.2014.6842355.
  20. Strashnov, E.V., 2020. Modelirovanie rezhimov poleta kvadrokoptera v sistemah virtual'nogo okruzheniya [Simulation of flight modes of a quadcopter in virtual environment systems]. Informatsionnye tekhnologii i Vychislitel'nye Sistemy [Information Technologies and Computing Systems], 1, pp.85–94. (in Russian).