Choosing a Camera for 3D Mapping

Choosing a Camera for 3D Mapping

Mykhailo Akopov, Svitlana Maksymova, Vladyslav Yevsieiev

Department of Computer-Integrated Technologies, Automation and Robotics,

Kharkiv National University of Radio Electronics, Ukraine

Abstract:

This paper examines the problem of three-dimensional mapping of space.. Creating 3D maps is an extremely urgent task. This is often done through mapping using a UAV. However, in order to create a technical map, it is necessary to select suitable sensors. This article provides an overview of sensors that can be used to create such maps. A pair of cameras has been selected, as well as a means of communication between them.

Key words: Unmanned aerial vehicles, Sensor, 3D map, Mapping, Cartography

Introduction

Since the invention of the first unmanned aerial vehicles (UAV) in 1933 until today, drones have become a part of everyday life. They have come in a great diversity of several applications such as military, construction, image and video mapping, medical, search and rescue, parcel delivery, hidden area exploration, oil rigs and power line monitoring, precision farming, wireless communication and aerial surveillance and so on [1]. From film production to military applications, UAVs are more cost-effective than their manned counterparts. UAVs are highly utilized in a wide range of services such as photography, path planning, search and rescue, inspection of power lines and civil constructions, etc. [2]. The obvious advantages of UAV data are their high spatial resolution and flexibility in acquisition and sensor integration [3]. Various methods and approaches can be used here [4]-[13].

Cartography remains one of the current areas for the use of unmanned aerial vehicles.

It should be noted that mapping can be more complex through the use of drones. that is, maps can have depth. That is, it becomes possible to create a 3D map. Such maps allow objects moving along it, for example, mobile robots, to choose optimal, albeit complex in trajectory, routes.

However, to build this type of map it is necessary to carefully select the sensors and devices used.

Thus, in this work we present a selection of necessary devices for constructing a 3D map.

Related works

Modern UAVs use a variety of sensors. Their choice is primarily determined by the tasks being solved. UAV and the sensors they can be equipped with are becoming more technologically advanced and common place [14] within different industry fields.

In [15] scientists consider photogrammetry and LiDAR using for application in forestry, and also the appropriate sensors that must be used in each situation are described. Paper [16] focused on the applications of UAVs based remote sensing of different traits with different phenotyping sensors. In this review, the UAVs platforms and the phenotyping sensors were briefly introduced. Researchers developed their own system that is based on lidars [17]. Their high-precision lidar odometry system allows achieving robust and real-time operation under challenging perceptual conditions. Ilci, V., & Toth, C. [18] assess the performance potential of autonomous vehicle sensor systems to obtain high-definition maps based on only using Velodyne sensor data for creating accurate point clouds. In article [19] two types of LiDAR sensor data (2D and 3D) are provided as well as navigation sensor data with commercial-level accuracy and high-level accuracy are desribed. In addition, two levels of sensor data are provided for the purpose of assisting in the complete validation of algorithms using consumer-grade sensors. Authors in [20] propose a new deep architecture for fusing camera and LiDAR sensors for 3D object detection. They note that due to the fact that the camera and LiDAR sensor signals have different characteristics and distributions, fusing these two modalities is expected to improve both the accuracy and robustness of 3D object detection.

Chang, M. F. and co-authors [21] use cameras and lidars. Their sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forwardfacing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. In [22] researchers note that LiDAR systems are rather expensive in comparison with cameras and other sensors. They analyze a variety of camera-only methods, where features are differentiably “lifted” from the multi-camera images onto the 2D ground plane, yielding a “bird's eye view” (BEV) feature representation of the 3D space around the vehicle. Scientists in [23] focus on vision-LiDAR approaches, whereas such a fusion would have many advantages. They write that hybridized solutions offer improvements in the performance of simultaneous localization and mapping, especially with respect to aggressive motion, lack of light, or lack of visual features. The paper by Zhong, X., & et al. [24] addresses the problem of achieving large-scale 3D reconstruction using implicit representations built from 3D LiDAR measurements.

Plenty of researchers propose to couple lidars with else sensors and/or methods. In [25] a tightly coupled lidar-IMU fusion method is introduced. The authors note that by jointly minimizing the cost derived from lidar and IMU measurements, the lidarIMU odometry (LIO) can perform well with considerable drifts after long-term experiment, even in challenging cases where the lidar measurement can be degraded.

It would take a very long time to list the achievements of scientists in the field of constructing 3D maps, but this is not the purpose of this article. Next, we will present our selection of necessary devices for further construction of 3D maps for navigation of mobile devices.

Sensors for obtaining a depth map

A lot of work considers the selection of a suitable camera for solving given problems [26]-[34].

Structured Light camera.

Let's start with, perhaps, one of the simplest, oldest and relatively cheap ways to measure depth - structured light. This method appeared essentially as soon as digital cameras appeared, i.e. more than 40 years ago and greatly simplified a little later, with the advent of digital projectors. The basic idea is extremely simple. We place next to a projector that creates, for example, horizontal (and then vertical) stripes and next to a camera that shoots a picture with stripes, as shown in Figure 1.

Figure 1: Structured Light Camera Operating Scheme

Since the camera and projector are offset relative to each other, the stripes will also move in proportion to the distance to the object. By measuring this displacement we can calculate the distance to the object. This scheme is hardly suitable for use on UAVs since, due to sunlight, the designed grid will be illuminated and unreadable for sensors.

Time of Flight camera.

The next way to get depth is more interesting. It is based on measuring the roundtrip delay of light (ToF - Time-of-Flight). As you know, the speed of modern processors is high, but the speed of light is low. In one clock cycle of a 3 GHz processor, light can travel only 10 centimeters. Or 10 clock cycles per meter. In fact, we need to measure the delay with which light returns to each point, this is presented in Figure 2.

Figure 2: Scheme of Operation of the Time of Flight Camera

The problems remain the same. There is a high probability of the image being exposed to bright light.

Camera based on lidar technology.

The first lidars (from LIDaR - Light Identification Detection and Ranging), built as bundles of similar devices rotating around a horizontal axis, were the first to be used by the militaries, then tested in car autopilots. They performed quite well there, which caused a powerful surge in investment in the region. Initially, the lidars rotated, giving a similar picture several times per second. Example of lidar camera performance is shown on Figure 3.

Figure 3: Example of Lidar Camera Performance

The problem with these devices can be considered their relatively high cost. Putting together a compact and inexpensive device is quite problematic. In addition, bright sunlight reflected from surfaces may cause incorrect data reading.

Light Field depth camera

The topic of plenoptics (from the Latin plenus - complete and optikos - visual) or light fields is still relatively little known to the general public, although professionals have begun to study it extremely actively. The main idea is to try to capture not just light at each point, but a two-dimensional array of light rays making each frame fourdimensional. Light field camera operation scheme is presented on Figure 4. In practice this is done using a microlens array.

Figure 4: Light Field Camera Operation Scheme

The main disadvantages of plenoptics can be considered the relatively low resolution and complexity of software implementation.

Depth from Stereo camera

Of the 5 methods under consideration for constructing depth video, only two - this and the previous one (stereo and plenoptics) - do not interfere with the sun and are not interfered with by surface reflections. At the same time, plenoptics is many times more expensive and less accurate at long distances. Depth from stereo - in terms of equipment cost - is the cheapest way to obtain depth, since cameras are now inexpensive and continue to become cheaper quickly. Example of depth from stereo camera operation is shown on Figure 5.

Figure 5: Example of Depth from Stereo Camera Operation

The difficulty is that further processing is much more resource-intensive than other methods. Despite the disadvantages discussed above, this technology was chosen as depth sensors for UAVs.

Waveshare IMX219-83

This stereo camera (Figure 6) was chosen due to its high resolution of 3280 x 2464, compactness, and the presence of additional sensors - accelerometer, gyroscope, magnetometer.

Figure 6: Stereo Camera Waveshare IMX219-83

To create a stereo pair, two compatible Pi cameras would be suitable too, for example the Raspberry Pi Camera Module 3 camera (Figure 7).

Figure 7: Raspberry Pi Camera Module 3

But in this case, it would be necessary to synchronize the cameras, which would add extra load to the computing module. To perform the calculations, Rasberry Pi CM4 and Carrier board were chosen for it.

Conclusion

In the modern world, UAVs are indispensable and are used in various fields of industry. UAVs are widely used to construct three-dimensional maps, which can subsequently be used to navigate cars, mobile robots, and other moving devices.

This paper discusses various sensors that can be used for UAVs for the purpose of mapping space. Various requirements are placed on sensors installed on UAVs. Among them, it should be noted their weight, as well as cost. To build a 3D map, the quality of the resulting three-dimensional image, as well as the complexity of its processing, is very important.

Thus, various types of sensors that may be suitable for the implementation of a system for building a 3D model of the environment of a mobile robot were considered. As a result of the analysis, a pair of cameras was selected, as well as a means of switching them.

References:

1. Mohsan, S. A. H., & et al. (2022). Towards the unmanned aerial vehicles (UAVs): A comprehensive review. Drones, 6(6), 147.

2. Ahmed, F., & et al. (2022). Recent advances in unmanned aerial vehicles: a review. Arabian Journal for Science and Engineering, 47(7), 7963-7984.

3. Yao, H., & et al. (2019). Unmanned aerial vehicle for remote sensing applications—A review. Remote Sensing, 11(12), 1443.

4. Rabotiahov, A., Kobylin, O., Dudar, Z., & Lyashenko, V. (2018, February). Bionic image segmentation of cytology samples method. In 2018 14th International Conference on Advanced Trends in Radioelecrtronics, Telecommunications and Computer Engineering (TCSET) (pp. 665-670). IEEE.

5. Lyashenko, V. V., Lyubchenko, V. A., Ahmad, M. A., Khan, A., & Kobylin, O. A. (2016). The Methodology of Image Processing in the Study of the Properties of Fiber as a Reinforcing Agent in Polymer Compositions. International Journal of Advanced Research in Computer Science, 7(1), 15-18.

6. Гиренко, А. В., Ляшенко, В. В., Машталир, В. П., & Путятин, Е. П. (1996). Методы корреляционного обнаружения объектов. Харьков: АО “БизнесИнформ, 112.

7. Al-Sherrawi, M. H., Lyashenko, V., Edaan, E. M., & Sotnik, S. (2018). Corrosion as a source of destruction in construction. International Journal of Civil Engineering and Technology, 9(5), 306-314.

8. Lyashenko, V., Ahmad, M. A., Sotnik, S., Deineko, Z., & Khan, A. (2018). Defects of communication pipes from plastic in modern civil engineering. International Journal of Mechanical and Production Engineering Research and Development, 8(1), 253-262.

9. Baker, J. H., Laariedh, F., Ahmad, M. A., Lyashenko, V., Sotnik, S., & Mustafa, S. K. (2021). Some interesting features of semantic model in Robotic Science. SSRG International Journal of Engineering Trends and Technology, 69(7), 38-44.

10. Abu-Jassar, A. T., Al-Sharo, Y. M., Lyashenko, V., & Sotnik, S. (2021). Some Features of Classifiers Implementation for Object Recognition in Specialized Computer systems. TEM Journal: Technology, Education, Management, Informatics, 10(4), 1645-1654.

11. Al-Sharo, Y. M., Abu-Jassar, A. T., Sotnik, S., & Lyashenko, V. (2021). Neural Networks As A Tool For Pattern Recognition of Fasteners. International Journal of Engineering Trends and Technology, 69(10), 151-160.

12. Sotnik, S., Mustafa, S. K., Ahmad, M. A., Lyashenko, V., & Zeleniy, O. (2020). Some features of route planning as the basis in a mobile robot. International Journal of Emerging Trends in Engineering Research, 8(5), 2074-2079.

13. Ahmad, M. A., Sinelnikova, T., Lyashenko, V., & Mustafa, S. K. (2020). Features of the construction and control of the navigation system of a mobile robot. International Journal of Emerging Trends in Engineering Research, 8(4), 1445-1449.

14. Olson, D., & Anderson, J. (2021). Review on unmanned aerial vehicles, remote sensors, imagery processing, and their applications in agriculture. Agronomy Journal, 113(2), 971-992.

15. Guimarães, N., & et al. (2020). Forestry remote sensing from unmanned aerial vehicles: A review focusing on the data, processing and potentialities. Remote Sensing, 12(6), 1046.

16. Feng, L., & et al. (2021). A comprehensive review on recent applications of unmanned aerial vehicle remote sensing with various sensors for high-throughput plant phenotyping. Computers and electronics in agriculture, 182, 106033.

17. Palieri, M., & et al. (2020). Locus: A multi-sensor lidar-centric solution for high-precision odometry and 3d mapping in real-time. IEEE Robotics and Automation Letters, 6(2), 421-428.

18. Ilci, V., & Toth, C. (2020). High definition 3D map creation using GNSS/IMU/LiDAR sensor integration to support autonomous vehicle navigation. Sensors, 20(3), 899.

19. Jeong, J., & et al. (2019). Complex urban dataset with multi-level sensors from highly diverse urban environments. The International Journal of Robotics Research, 38(6), 642-657.

20. Yoo, J. H., & et al. (2020). 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, Springer International Publishing, 720-736.

21. Chang, M. F., & et al. (2019). Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8748-8757.

22. Harley, A. W., & et al. (2023). Simple-BEV: What really matters for multi-sensor bev perception?. In 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2759-2765.

23. Debeunne, C., & Vivet, D. (2020). A review of visual-LiDAR fusion based simultaneous localization and mapping. Sensors, 20(7), 2068.

24. Zhong, X., & et al. (2023). Shine-mapping: Large-scale 3d mapping using sparse hierarchical implicit neural representations. In 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 8371-8377.

25. Ye, H., & et al. (2019). Tightly coupled 3d lidar inertial odometry and mapping. In 2019 International Conference on Robotics and Automation (ICRA),

IEEE, 3144-3150.

26. Lyashenko, V., & et al. (2023). Automated Monitoring and Visualization System in Production. Int. Res. J. Multidiscip. Technovation, 5(6), 09-18.

27. I. Nevliudov, & et al. (2022). Object Recognition for a Humanoid Robot Based on a Microcontroller. In IEEE XVIII International Conference on the Perspective Technologies and Methods in MEMS Design (MEMSTECH), IEEE, 61-64.

28. Yevsieiev, V., & et al. (2022). Software Implementation Concept Development for the Mobile Robot Control System on ESP-32CAM In Current issues of science, prospects and challenges: collection of scientific papers «SCIENTIA» II International Scientific and Theoretical Conference, 2, Sydney, Australia: European Scientific Platform, 54-56.

29. Maksymova, S., & Akopov, M. (2023). Selection of Sensors for Building a 3D Model of the Mobile Robot's Environment. In VII International Conference Manufacturing & Mechatronic Systems (M&MS), 33-35.

30. Nikitin, V., & et al. (2023). Traffic Signs Recognition System Development. Multidisciplinary Journal of Science and Technology, 3(3), 235-242.

31. Lyashenko, V., & Sotnik, S. (2020). Analysis of Basic Principles for Sensor System Design Process Mobile Robots. Journal La Multiapp, 1(4), 1-6.

32. Mustafa, S. K., Kopot, M., Ahmad, M. A., Lyubchenko, V., & Lyashenko, V. (2020). Interesting applications of mobile robotic motion by using control algorithms. International Journal of Advanced Trends in Computer Science and Engineering, 9(3), 3847-3852.

33. Abu-Jassar, A. T., Attar, H., Lyashenko, V., Amer, A., Sotnik, S., & Solyman, A. (2023). Access control to robotic systems based on biometric: the generalized model and its practical implementation. International Journal of Intelligent Engineering and Systems, 16(5), 313-328.

34. Al-Sharo, Y. M., Abu-Jassar, A. T., Sotnik, S., & Lyashenko, V. (2023). Generalized Procedure for Determining the Collision-Free Trajectory for a Robotic Arm. Tikrit Journal of Engineering Sciences, 30(2), 142-151.