JP6473188B2 - Method, apparatus and program for generating depth map - Google Patents

Description

  Unmanned aerial vehicles such as unmanned aerial vehicles (UAVs) can be used to perform surveillance, reconnaissance, and search duties for a wide variety of military and civilian applications. In some cases, the UAV may carry a camera for generating image data of an object in certain circumstances. Based on the specific application, a camera calibration procedure may be performed to determine one or more parameters of the camera.

  Existing approaches for camera calibration may not be optimal in some cases. For example, some camera calibration methods may not be optimal for calibrating an imaging system that includes multiple cameras.

  The present disclosure provides systems, methods, and devices for multiple cameras and other types of imaging devices with improved calibration. In some embodiments, the plurality of calibration techniques described herein are obtained by one or more imaging devices to be calibrated comprising a plurality of features and one or more identifiable reference markers. Use multiple images. The plurality of reference markers can be used to facilitate association between a plurality of image coordinates of a plurality of features in a plurality of images and a plurality of global coordinates of a corresponding plurality of features on the calibration target. Based on the determined correspondence, one or more calibration parameters (eg, multiple external and / or internal parameters) for multiple imaging devices may be calculated. Some aspects of the present disclosure determine a plurality of calibration parameters using a plurality of images that depict only a portion of the entire calibration object (eg, only some of all of the plurality of features on the calibration object). Is possible. Advantageously, this approach provides improved accuracy and flexibility of the calibration process, particularly with respect to calibration of multiple imaging devices.

Accordingly, in one aspect, a method for calibrating one or more imaging devices is provided. The method is
Receiving, from each of the one or more imaging devices, a plurality of images to be calibrated comprising a plurality of features arranged in a repeating pattern and one or more reference markers;
Determining the spatial relationship between one or more reference markers in each of the plurality of images and one or more reference markers on the calibration object with the assistance of one or more processors;
Based on the spatial relationship and with the assistance of one or more processors, a plurality of image coordinates of the features in each image of the plurality of images and a plurality of global coordinates for the corresponding features on the calibration object Formulating the correspondence between,
Determining a plurality of calibration parameters for one or more imaging devices based on the correspondence and with the assistance of one or more processors, and
Each of the one or more reference markers is uniquely identifiable in each image of the plurality of images;
The plurality of images includes at least one image that captures only some of the plurality of features.

  In some embodiments, the one or more imaging devices are mounted by an unmanned aerial vehicle.

  In some embodiments, each image of the plurality of images captures one or more reference markers and at least a subset of the plurality of features on the calibration object. Multiple images from each of the one or more imaging devices may be taken at different positions and orientations relative to the calibration object.

  In some embodiments, the calibration object is a checkerboard that includes a plurality of squares, and the plurality of features includes a plurality of corner points of the plurality of squares. Each of the one or more reference markers may be positioned in one of the plurality of squares. The plurality of features may include four outer corners of the checkerboard and the at least one image may exclude at least one of the four outer corners.

  In some embodiments, the one or more reference markers includes at least three reference markers. The one or more reference markers may include a plurality of circular dots, triangles, or quadrilaterals. One or more reference markers may be superimposed on the repeating pattern. The one or more reference markers may form irregular polygonal vertices on the calibration object. The one or more fiducial markers can include three fiducial markers that form a plurality of vertices of an isosceles or unequal triangle.

  In some embodiments, the one or more fiducial markers form a plurality of vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes the irregular polygon in the plurality of images. Identifying correspondence between the plurality of vertices and the plurality of irregular polygon vertices on the calibration object. The features may be arranged in a repeating pattern on the plane, and the one or more reference markers may form a shape that is non-rotational symmetric in the plane.

  In some embodiments, the one or more imaging devices include a first camera and a second camera that capture multiple images at substantially the same time. The formulating step may include identifying a plurality of features present in both the image obtained by the first camera and the corresponding image obtained by the second camera. The plurality of calibration parameters may include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices. The plurality of calibration parameters may include the position and orientation of each of the one or more imaging devices relative to the calibration object.

In another aspect, a system for calibrating one or more imaging devices is provided. the system,
One or more imaging devices each configured to capture a plurality of images to be calibrated including a plurality of features arranged in a repeating pattern and one or more reference markers;
With one or more processors,
Each of the one or more reference markers is uniquely identifiable in each image of the plurality of images;
The plurality of images includes at least one image that captures only a portion of all of the plurality of features,
One or more processors may be collectively or individually;
Receiving a plurality of images from each of one or more imaging devices;
Determining a spatial relationship between one or more reference markers in each image of the plurality of images and one or more reference markers on the calibration object;
Based on a plurality of predetermined global coordinates of one or more reference markers, a plurality of image coordinates of a plurality of features in each image of a plurality of images, and a plurality of global coordinates relating to a plurality of corresponding features on the calibration target Formulate the correspondence between and
Based on the correspondence, the plurality of calibration parameters for the one or more imaging devices are determined.

  In some embodiments, the one or more imaging devices are mounted by an unmanned aerial vehicle.

  In some embodiments, each image of the plurality of images captures one or more reference markers and at least a subset of the plurality of features on the calibration object. Multiple images from each of the one or more imaging devices may be taken at different positions and orientations relative to the calibration object.

  In some embodiments, the calibration object is a checkerboard that includes a plurality of squares, and the plurality of features includes a plurality of corner points of the plurality of squares. Each of the one or more reference markers may be positioned in one of the plurality of squares. The plurality of features may include four outer corners of the checkerboard and the at least one image may exclude at least one of the four outer corners.

  In some embodiments, the one or more reference markers includes at least three reference markers. The one or more reference markers may include a plurality of circular dots, triangles, or quadrilaterals. One or more reference markers may be superimposed on the repeating pattern. The one or more reference markers may form irregular polygonal vertices on the calibration object. The one or more fiducial markers can include three fiducial markers that form a plurality of vertices of an isosceles or unequal triangle.

  In some embodiments, the one or more fiducial markers form a plurality of vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes the irregular polygon in the plurality of images. Identifying a correspondence between a plurality of vertices and a plurality of irregular polygon vertices on the calibration object. The features may be arranged in a repeating pattern on the plane, and the one or more reference markers may form a shape that is non-rotational symmetric in the plane.

  In some embodiments, the one or more imaging devices include a first camera and a second camera that capture multiple images at substantially the same time. The formulating step may include identifying a plurality of features present in both the image obtained by the first camera and the corresponding image obtained by the second camera. The plurality of calibration parameters may include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices. The plurality of calibration parameters may include the position and orientation of each of the one or more imaging devices relative to the calibration object.

In another aspect, a method is provided for generating a plurality of depth maps using one or more imaging devices. The method is
Obtaining one or more images from each of the one or more imaging devices;
Processing one or more images from each of the one or more imaging devices based on a plurality of calibration parameters for the one or more imaging devices to generate a three-dimensional depth map with the assistance of one or more processors. And
The plurality of calibration parameters are determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices,
The calibration object includes a plurality of features arranged in a repeating pattern and one or more reference markers,
Each of the one or more reference markers is uniquely identifiable within each calibration image of the plurality of calibration images;
The plurality of calibration images includes at least one calibration image that captures only a portion of all of the plurality of features.

  In some embodiments, the one or more imaging devices are mounted by an unmanned aerial vehicle. The depth map may include information indicating multiple distances of one or more objects from the unmanned aerial vehicle.

  In some embodiments, each calibration image of the plurality of calibration images captures one or more reference markers and at least a subset of the plurality of features on the calibration object. A plurality of calibration images from each of a plurality of imaging devices may be taken at different positions and orientations with respect to the calibration object.

  In some embodiments, the calibration object is a checkerboard that includes a plurality of squares, and the plurality of features includes a plurality of corner points of the plurality of squares. Each of the one or more reference markers may be positioned in one of the plurality of squares. The features may include four outer corners of the checkerboard, and the at least one calibration image may exclude at least one of the four outer corners.

  In some embodiments, the one or more reference markers includes at least three reference markers. The one or more reference markers may include a plurality of circular dots, triangles, or quadrilaterals. One or more reference markers may be superimposed on the repeating pattern. The one or more reference markers may form irregular polygonal vertices on the calibration object. The one or more fiducial markers can include three fiducial markers that form a plurality of vertices of an isosceles or unequal triangle.

  In some embodiments, the one or more reference markers form a plurality of vertices of an irregular polygon in the plurality of images, and the plurality of calibration parameters are at least the irregular polygon in the plurality of calibration images. And the correspondence between the vertices of the irregular polygon on the object to be calibrated. The features may be arranged in a repeating pattern on the plane, and the one or more reference markers may form a shape that is non-rotational symmetric in the plane.

  In some embodiments, the one or more fiducial markers form a plurality of vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes the irregular polygon in the plurality of images. Identifying correspondence between the plurality of vertices and the plurality of irregular polygon vertices on the calibration object. The features may be arranged in a repeating pattern on the plane, and the one or more reference markers may form a shape that is non-rotational symmetric in the plane.

  In some embodiments, the one or more imaging devices include a first camera and a second camera that capture multiple images at substantially the same time. Processing the one or more images may include identifying features present in both the image obtained by the first camera and the corresponding image obtained by the second camera. Processing the one or more images may include determining a parallax between an image captured by the first camera and a corresponding image captured by the second camera. The plurality of calibration parameters may include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices. The plurality of calibration parameters may include the position and orientation of each of the one or more imaging devices relative to the calibration object. Processing the one or more images may include reducing optical distortion in at least one of the one or more images.

In another aspect, a system is provided for generating a plurality of depth maps using one or more imaging devices. the system,
One or more imaging devices each configured to obtain a plurality of images;
One or more processors, and
One or more processors may be collectively or individually;
Obtaining one or more images from each of the one or more imaging devices;
Configured to process one or more images from each of the one or more imaging devices based on a plurality of calibration parameters for the one or more imaging devices to generate a three-dimensional depth map;
The plurality of calibration parameters are determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices,
The calibration object includes a plurality of features arranged in a repeating pattern and one or more reference markers,
Each of the one or more reference markers is uniquely identifiable within each calibration image of the plurality of calibration images;
The plurality of calibration images includes at least one calibration image that captures only a portion of all of the plurality of features.

  In some embodiments, the one or more imaging devices are mounted by an unmanned aerial vehicle. The depth map may include information indicating multiple distances of one or more objects from the unmanned aerial vehicle.

  In some embodiments, each calibration image of the plurality of calibration images captures one or more reference markers and at least a subset of the plurality of features on the calibration object. A plurality of calibration images from each of a plurality of imaging devices may be taken at different positions and orientations with respect to the calibration object.

  In some embodiments, the calibration object is a checkerboard that includes a plurality of squares, and the plurality of features includes a plurality of corner points of the plurality of squares. Each of the one or more reference markers may be positioned in one of the plurality of squares. The features may include four outer corners of the checkerboard, and the at least one calibration image may exclude at least one of the four outer corners.

  In some embodiments, the one or more reference markers includes at least three reference markers. The one or more reference markers may include a plurality of circular dots, triangles, or quadrilaterals. One or more reference markers may be superimposed on the repeating pattern. The one or more reference markers may form irregular polygonal vertices on the calibration object. The one or more fiducial markers can include three fiducial markers that form a plurality of vertices of an isosceles or unequal triangle.

  In some embodiments, the one or more reference markers form a plurality of vertices of an irregular polygon in the plurality of images, and the plurality of calibration parameters are at least the irregular polygon in the plurality of calibration images. And the correspondence between the vertices of the irregular polygon on the object to be calibrated. The features may be arranged in a repeating pattern on the plane, and the one or more reference markers may form a shape that is non-rotational symmetric in the plane.

  In some embodiments, the one or more fiducial markers form a plurality of vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes the irregular polygon in the plurality of images. Identifying a correspondence between a plurality of vertices and a plurality of irregular polygon vertices on the calibration object. The features may be arranged in a repeating pattern on the plane, and the one or more reference markers may form a shape that is non-rotational symmetric in the plane.

  In some embodiments, the one or more imaging devices include a first camera and a second camera that capture multiple images at substantially the same time. Processing the one or more images may include identifying features present in both the image obtained by the first camera and the corresponding image obtained by the second camera. Processing the one or more images may include determining a parallax between an image captured by the first camera and a corresponding image captured by the second camera. The plurality of calibration parameters may include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices. The plurality of calibration parameters may include the position and orientation of each of the one or more imaging devices relative to the calibration object. Processing the one or more images may include reducing optical distortion in at least one of the one or more images.

  It will be appreciated that different aspects of the present invention may be recognized individually, collectively or in combination with each other. The various aspects of the invention described herein apply to any of the specific applications specified below, or for any other types of movable objects. obtain. Any description of an aircraft herein can be applied to and used for any movable object, such as any vehicle. Additionally, the systems, devices, and methods disclosed herein in the context of motion in the air (eg, flight) can be performed on the ground or on the water, below the surface, or in space. It can also be applied to other types of contexts related to movement, such as movement at. Furthermore, any description of a rotor or rotor assembly herein may apply to any propulsion system, device, or mechanism (eg, propeller, wheel, axle) that is configured to generate propulsion by rotation. Can be used for them.

Other objects and features of the present invention will become apparent upon review of the specification, the claims, and the accompanying drawings.
Include by reference

  All publications, patents, and patent applications mentioned in this specification are shown with each individual publication, patent, or patent application specifically and individually incorporated by reference. Are incorporated herein by reference to the same extent.

  The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention may be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which: Will be done.

2 shows a monocular camera model, according to embodiments.

Fig. 6 illustrates a model for mapping from a global coordinate system to an image coordinate system, according to embodiments.

Fig. 6 illustrates a model for mapping from a global coordinate system to an image coordinate system, according to embodiments.

Fig. 4 illustrates a calibration object suitable for use in calibration of an imaging device, according to embodiments.

Fig. 6 illustrates a plurality of images of a checkerboard calibration target according to embodiments.

FIG. 6 illustrates a plurality of reference markers that are uniquely identifiable based on relative arrangements according to embodiments. FIG. 6 illustrates a plurality of reference markers that are uniquely identifiable based on relative arrangements according to embodiments.

Fig. 6 illustrates a calibration object having a plurality of reference markers according to embodiments.

4 illustrates a method for calibrating one or more imaging devices using a calibration object having one or more reference markers, according to embodiments.

FIG. 7 illustrates identification of features to be calibrated and reference markers in an image according to embodiments. FIG. 7 illustrates identification of features to be calibrated and reference markers in an image according to embodiments. FIG. 7 illustrates identification of features to be calibrated and reference markers in an image according to embodiments. FIG. 7 illustrates identification of features to be calibrated and reference markers in an image according to embodiments.

FIG. 6 illustrates models of multiple spatial relationships between multiple coordinate systems in binocular camera calibration, according to embodiments. FIG.

Fig. 6 shows a model of mapping between a plurality of distortion image coordinates and distortion correction image coordinates, according to embodiments.

6 illustrates an image modification procedure for a binocular camera system, according to embodiments. 6 illustrates an image modification procedure for a binocular camera system, according to embodiments.

5 shows a plurality of binocular camera images obtained as a result of a process of performing image correction according to embodiments.

6 illustrates a method for generating a plurality of depth maps from image data, according to embodiments.

2 shows a UAV according to embodiments.

Fig. 6 illustrates a movable object including a support mechanism and a load according to embodiments.

1 illustrates a system for controlling a movable object, according to embodiments.

  The present disclosure provides improved systems, methods, and devices for performing calibration of multiple imaging devices such as cameras. The multiple approaches described herein may involve obtaining multiple images to be calibrated using one or more imaging devices. The calibration object may include a plurality of features and one or more reference markers. Correspondence between image coordinates of multiple features in multiple images and global coordinates of multiple features to be calibrated, even when some of the multiple images do not capture all of the multiple features to be calibrated In order to facilitate, one or more fiducial markers may be identifiable. This approach is accurate and reliable even in multiple situations where each of the multiple imaging devices has difficulty obtaining the entire image data to be calibrated, for example when multiple imaging devices are calibrated simultaneously. Can be advantageous to provide high calibration. Additionally, such multiple approaches allow the use of larger multiple calibration objects, which can improve the accuracy of the resulting multiple calibration parameters.

  The techniques described herein are for use in a wide variety of applications involving imaging and imaging data, such as vision-based navigation using one or more imaging devices mounted by moving objects. Can be implemented. By way of example, the systems, methods, and devices of the present disclosure may be used to calibrate one or more cameras that are mounted by an unmanned aerial vehicle (UAV). Each of the one or more cameras may be used to capture a plurality of images of a checkerboard calibration target having at least three circular fiducial markers superimposed on the checkerboard squares. Multiple reference markers can be uniquely identified in multiple images to facilitate the determination of multiple global coordinates of multiple corner points of the checkerboard corresponding to multiple corner points captured in the image data obtain. This correspondence information can be used to determine a plurality of calibration parameters for a plurality of UAV cameras. Subsequent image data captured by multiple cameras is based on multiple calibration parameters, for example, to generate multiple 3D (3D) depth maps of the surrounding environment and / or multiple objects in the environment. Can be processed.

  The techniques described herein can be used to calibrate one or more imaging devices (eg, cameras, scanners, etc.). An imaging device, which may also be referred to herein as a “vision sensor”, is configured to detect electromagnetic radiation (eg, visible light, infrared, and / or ultraviolet light) and generate image data based on the detected electromagnetic radiation. obtain. For example, the imaging device may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor that generates multiple electrical signals in response to multiple wavelengths of light. The resulting plurality of electrical signals can be processed to generate image data. The image data generated by the imaging device may include one or more images that may be a plurality of still images (eg, photographs), a plurality of moving images (eg, videos), or a suitable combination of these. The image data can be multicolor (eg, RGB, CMYK, HSV) or single color (eg, grayscale, black and white, sepia).

  Multiple imaging devices of the present disclosure may be carried by various types of objects, such as by one or more of the multiple movable objects described herein. The imaging device may be positioned on any suitable portion of the object, such as above, below, to the side, or within the object. Some imaging devices may be mechanically connected to the object such that the spatial arrangement and / or movement of the object corresponds to the spatial arrangement and / or movement of the plurality of imaging devices. The imaging device may be connected to the object via a rigid connection member so that it does not move relative to the part of the object to which the imaging device is attached. Alternatively, the connection between the imaging device and the object may allow movement of the imaging device relative to the object. The connection can be a permanent connection or a non-permanent (eg, releasable) connection. Suitable connection methods may include adhesives, bonding, welding, and / or fasteners (eg, screws, nails, pins, etc.). Optionally, the imaging device may be integrally formed with a part of the object. In addition, the imaging device allows data collected by the imaging device to be used for various functions of the object (eg, navigation, control, propulsion, communication with a user or other device) such as the embodiments described herein. Etc.) can be electrically connected to a part of the object (eg, processing unit, control system, data storage) to allow it to be used for.

  In some embodiments, multiple imaging devices are mounted by UAV. For example, the UAV can be a small UAV (eg, having a maximum dimension of 1.5 m or less, having a weight of 10 kg or less). A UAV may be a rotorcraft, such as a multi-rotor aircraft (eg, a quadcopter) that is propelled to move through the air by a plurality of propellers. The plurality of UAVs described herein may be fully autonomous, semi-autonomous (eg, by a suitable computing system such as an onboard controller), or (eg, by a human user). Can be operated manually. A UAV may receive multiple commands from a suitable entity (eg, a human user or an autonomous control system) and respond to such multiple commands by performing one or more operations. For example, a UAV takes off from the ground, moves in the air (eg, with up to 3 translational degrees of freedom, and with up to 3 rotational degrees of freedom), moves to a target location, or a series of target locations, hovers in the air , Landing on the ground, and other things can be controlled. As another example, the UAV is controlled to move at a specified speed and / or acceleration (eg, with up to 3 translational degrees of freedom and up to 3 rotational degrees of freedom) or along a specified path of travel. Can be done. Further, the plurality of commands may be used to control one or more UAV components (eg, sensors, actuators, propulsion units, loads, etc.), such as the components described herein.

  In some embodiments, the imaging device can be a camera. Although the specific embodiments provided herein are described in the context of a camera, the present disclosure can be applied to any suitable imaging device and is here related to multiple cameras. It will be appreciated that any description may apply to other types of imaging devices. A camera can be used to generate multiple 2D images of a 3D scene (eg, environment, one or more objects, etc.). The plurality of images generated by the camera may represent a projection of a 3D scene onto a 2D image plane. Thus, each point in the 2D image corresponds to 3D spatial coordinates in the scene.

FIG. 1 shows a model 100 of a single camera (monocular camera), according to embodiments. The model 100 includes a global coordinate system 102, a camera coordinate system 104, an image plane coordinate system 106, and a pixel coordinate system 108. Global coordinate system 102, which may also be referred to herein as a “global coordinate system”, represents a reference frame of the scene captured by the camera and / or the environment in which the camera is operating. The global coordinate system 102 has a coordinate origin O _w and a plurality of 3D coordinate axes X _w , Y _w , and Z _w . In some embodiments, the global coordinate system 102 may be defined with respect to an object of interest in a scene, such as a calibration object, as described in more detail below. The camera coordinate system 104 represents a reference frame of the camera, and has a coordinate origin O _c and a plurality of 3D coordinate axes X _c , Y _c , and Z _c . Axis Z _c is the direction toward the outside may correspond to the optical axis of the camera is a positive direction.

The image plane coordinate system 106 and the pixel coordinate system 108, which may be referred to as “image coordinate system” in this specification, represent a reference frame of an image generated by the camera. The image plane coordinate system has a coordinate origin O _p and a plurality of 2D coordinate axes X _p , Y _p . The plurality of positions of the plurality of pixels in the image can be represented in terms of a pixel coordinate system 108 having a coordinate origin at the upper left corner of the image and a plurality of 2D coordinate axes u and v. The u-axis can be parallel to multiple rows of the image, the right direction is a positive direction, and the v-axis can be parallel to multiple columns of the image, with the downward direction being the positive direction. In some embodiments, the camera coordinate system 104 may be aligned with the image plane coordinate system 106 and the pixel coordinate system 108. For example, the _Xc axis of the camera coordinate system 104 can be parallel to multiple rows of the image and can be aligned in the same direction as the u axis of the pixel coordinate system 108. Similarly, the _Yc axis of the camera coordinate system 104 may be parallel to multiple columns of the image and may be aligned in the same direction as the v axis of the pixel coordinate system 108.

  Although FIG. 1 depicts a single imaging device, the embodiments described herein can be one, two, three, four, five, or more imaging devices. It will be appreciated that it can be applied to any suitable number of imaging devices. The plurality of imaging devices can all be of the same type. Alternatively, at least some of the plurality of imaging devices may be of different types. Multiple imaging devices may be positioned and oriented as desired with respect to each other. For example, the various embodiments provided herein utilize two imaging devices (binocular vision), such as a left camera and a right camera. The plurality of imaging devices may be positioned at a plurality of different positions and orientations such that each imaging device has a different field of view. Multiple fields of view of multiple imaging devices can overlap. In some embodiments, each of the plurality of imaging devices provides image data of the same scene from a plurality of different positions and / or orientations. The multiple imaging devices may be configured to capture multiple images of the scene at the same time or approximately the same time. Alternatively, some imaging devices may be configured to capture multiple images at different times than other multiple imaging devices.

  In some embodiments, it may be desirable to obtain 3D information of a scene using 2D image data of the scene collected by one or more imaging devices. For example, various applications are based on multiple 2D images of a scene (eg, multiple images taken at different locations and orientations), and 3D reconstruction of the scene (eg, point cloud, occupancy grid, depth map, Generating a 3D model such as a parallax map). In embodiments where multiple imaging devices are used, as described in further detail below, 3D reconstruction is an image obtained by multiple imaging devices positioned at different locations and / or orientations. It can be generated based on multiple parallaxes between data. Alternatively or in combination, 3D information can also be recovered from image data obtained from a single imaging device, eg, using structure-from-motion or other suitable techniques. The resulting 3D reconstruction is based on various types of quantitative information about the scene, eg, multiple spatial positions of multiple objects (eg, camera, distance from other objects), multiple multiple of objects It can be used to determine dimensions (eg, length, width, height). This information can be useful in a wide variety of applications, such as computer vision, object recognition, tracking, posture estimation, self-motion determination, and the like. For example, in the context of UAV operation, scene reconstruction may be useful for multiple functionalities such as navigation, obstacle avoidance, mapping, object tracking, and the like.

  In some embodiments, the multiple scene reconstruction techniques described herein may include multiple image coordinates in an image of a scene and multiple global coordinates (also known as global coordinates) of an actual scene. Entails determining the correspondence between. This correspondence can be identified by determining multiple transformations or multiple mappings between various coordinate systems, eg, global coordinate system, camera coordinate system, and image coordinate system. The plurality of mathematical relationships that define these mappings may be determined based on a scene, a plurality of imaging devices, and a plurality of models of image planes, such as a pinhole camera model.

  FIG. 2 illustrates a model 200 for mapping from a global coordinate system to an image coordinate system, according to embodiments. The mapping model 200 depicts multiple relationships between the world, the camera, and the image coordinate system. The mapping model 200 includes an imaging process (eg, projecting a plurality of global coordinates of an object in the scene onto a plurality of image coordinates in the image) and a reconstruction process (eg, using a plurality of image coordinates in the image in the scene). Can be used to describe the restoration of multiple global coordinates of an object).

Global coordinate system is a mathematical formula
Can be associated with the camera coordinate system. Here, R is a rotation matrix, T is a translation vector, and indicates how the camera coordinate system is oriented and positioned with respect to the global coordinate system. The plurality of camera coordinates can be scaled to obtain a corresponding plurality of image plane coordinates.

In embodiments where the camera has optical lens distortion (eg, radial distortion, eccentric distortion, thin prism distortion), the plurality of image plane coordinates with distortions X _d , Y _d are: Can be determined from a plurality of image plane coordinates using a number of techniques known to those skilled in the art, such as from a mathematical equation relating to
here,
And k ₁ , k ₂ , k ₃ , p ₁ , p ₂ , s ₁ , s ₂ are distortion parameters k ₁ , k ₂ , k ₃ are radial distortion parameters, and p ₁ , p ₂ are eccentricity And s ₁ and s ₂ are thin prism distortion parameters). In some embodiments, only four of the plurality of parameters (eg, k ₁ , k ₂ , p ₁ , p ₂ ) are calculated.

A plurality of pixel coordinates can be obtained using the following relationship.
Where M is the camera internal matrix, u ₀ , v ₀ are the coordinates of the principal point (also known as the image center, where the optical axis intersects the image plane),
Where f is the focal length of the camera, and a _x and a _y are the pixel dimensions in the horizontal and vertical directions, respectively.

FIG. 3 illustrates a model 250 for mapping from a global coordinate system to an image coordinate system, according to embodiments. The plurality of relationships among the global coordinate system, camera coordinate system, distortion corrected image plane coordinate system, distortion image plane coordinate system, and pixel coordinate system are similar to those described herein with respect to the mapping model 200 of FIG. It can be. Similar to the mapping model 200, the mapping model 250 describes the imaging process (eg, projecting multiple global coordinates ( _Xw , _Yw , _Zw ) to multiple pixel coordinates (u, v)). Can be used. A plurality of global coordinates of the object are subjected to image distortion correction to obtain a plurality of pixel coordinates (u ′, v ′) corrected for distortion, and then a plurality of global coordinates using the plurality of pixel coordinates corrected for distortion. Can be reconstructed from multiple pixel coordinates.

In order to accurately perform scene reconstruction from image data, it is beneficial to determine or estimate one or more parameters of multiple imaging devices associated with multiple mappings described herein. It may be. These parameters include a plurality of external parameters (rotation matrix R and translation vector T indicating how each imaging device is positioned and oriented with respect to the calibration object) and a plurality of internal parameters (focal length, The position of the principal point, an internal matrix M containing information on the pixel size, optical distortion parameters k ₁ , k ₂ , k ₃ , p ₁ , p ₂ , s ₁ , s ₂ ). In embodiments where multiple imaging devices are used, the multiple external parameters are how the multiple imaging devices are positioned and oriented relative to each other, as described in more detail below. Can also include a plurality of rotation matrices and translation vectors for converting between the plurality of camera coordinate systems of the plurality of imaging devices. The process of obtaining a plurality of external and / or internal parameters of one or more imaging devices may be referred to herein as “camera calibration”.

  Camera calibration may be particularly useful when multiple imaging devices are mounted on or mounted on a movable object such as a UAV. Such movement of multiple movable objects can cause multiple changes in position and / or orientation of multiple imaging devices due to, for example, multiple vibrations and other movements that can affect calibration parameters. . Calibration of multiple imaging devices mounted by a movable object may occur while the movable object is operating (online calibration) and / or while the movable object is not operating (offline calibration). For example, offline calibration for a plurality of cameras mounted by the UAV may be performed when the UAV is turned off and / or the UAV is not in flight, while online calibration is performed when the UAV is powered Can be executed while the UAV is on and / or the UAV is flying.

  Camera calibration can be performed in various ways. In some embodiments, the calibration process obtains a plurality of images of a scene having a plurality of features, and then a plurality of feature image coordinates in the plurality of images and a corresponding plurality of feature global coordinates in the scene. With identifying multiple correspondences between Thereafter, a plurality of calibration parameters (eg, defined based on a plurality of mapping models described herein) are based on a plurality of identified correspondences and mappings between the global coordinate system and the image coordinate system. Can be estimated. For example, monocular calibration (determining multiple parameters of a single camera) generates an estimate of multiple calibration parameters (eg, R, T, M) based on a pinhole camera model, and then distortion Estimating the impact (eg, multiple initial values of multiple distortion parameters are assumed to be zero), and then obtaining the optimal multiple camera calibration parameters (eg, using the Levenberg-Markert algorithm) Can be used) to perform parameter optimization.

  In some embodiments, calibration objects are used to facilitate identification of multiple correspondences of world-image coordinates. The calibration object may be any structure having a defined arrangement feature. For example, the calibration object can be 1D (eg, linear, curved, or curved shape), 2D (eg, planar or other flat surface), or 3D (eg, cube, sphere, prism, 2 or more) Intersecting planes). The features can be easily identified from the image data (eg, using multiple image recognition algorithms, multiple feature extraction algorithms, or other computer vision techniques) and other than features (eg, , Other parts to be calibrated, other objects in the scene, image noise). In some embodiments, the plurality of features are a plurality of points to be calibrated (eg, a plurality of corners, vertices, intersections) (also referred to herein as “feature points”). Alternatively, or in combination, the features may include multiple lines (eg, linear, curved, or curved), multiple edges (eg, between two differently colored areas), multiple It can be a geometric shape (eg, circle, triangle, quadrilateral, etc.), symbol or letter (eg, alphanumeric), or any suitable combination thereof. Additionally, multiple characteristics (eg, size, shape, color) of each feature can be varied as desired.

  A calibration object may include any suitable number of features and combinations thereof. For example, the calibration object may include at least 10, 20, 30, 40, 50, 60, 70, 80, 90 or 100 features. By using a relatively large number of features, the accuracy of the calibration process can be improved. In some embodiments, the calibration object may include only one type of feature, and in other embodiments, the object may be at least 2, 3, 4, 5, or more different types of features, etc. It may contain more than one type of feature. The features can be positioned at a plurality of defined locations on the calibration target such that the position and / or orientation of each feature relative to the calibration target is known. In some embodiments, the features have a repeating pattern (eg, multiple rows and columns) having a given plurality of dimensions (eg, length, width, height, spacing between features). (Lattice) on the object to be calibrated. The features can be static features so that their position and / or orientation on the calibration object does not change. Alternatively, the features may be dynamic features that change position and / or orientation. The features described herein may be integrally formed with the calibration object (eg, printed or applied to a portion of the object) or provided separately and connected to the object (E.g., attached or affixed to a portion of the subject).

  In some embodiments, the calibration process involves capturing a set of images to be calibrated by each imaging device to be calibrated. Each image data set may include at least 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, or more images to be calibrated. The plurality of images may depict the calibration object at different positions and / or orientations. Each image can be processed to identify multiple features present in the image and determine their image coordinates. A plurality of correspondences between the plurality of image coordinates of the plurality of features in the image data and the plurality of global coordinates of the plurality of features on the calibration object may be determined, followed by (for example, as described herein) Can be used to solve a plurality of calibration parameters (based on a plurality of mathematical relationships). Optionally, in embodiments where multiple imaging devices are used to obtain multiple image data sets, multiple correspondences between multiple image coordinates of multiple features across multiple image data sets may also be determined, It may be used to estimate a plurality of parameters for a plurality of imaging devices (eg, position and / or orientation of a plurality of imaging devices relative to each other).

  FIG. 4 illustrates a calibration object 300 suitable for use in calibrating an imaging device, according to embodiments. Object 300 comprises a planar checkerboard pattern having a repeating pattern of a plurality of black squares 302 alternating with a plurality of white squares 304. Each of the plurality of squares 302, 304 may have the same size and be arranged in a grid having a plurality of rows and columns. In some embodiments, the plurality of corner points of the plurality of squares (eg, as shown in the plurality of circles 305) serve as feature points of the calibration object 300 to perform the imaging device calibration. Bear. The plurality of corner points may be relatively easy to distinguish from other portions of the image data using a plurality of machine vision methods. Alternatively or in combination, other portions of the checkerboard may serve as features of the object 300 for calibration purposes (eg, black square 302, white square 304, etc.). Multiple features of the object 300 can be identified from the image data of the calibration object 300 using various feature extraction techniques, and multiple image coordinates of each feature can be determined. This process can be facilitated by knowledge of the geometry of the calibration object 300. For example, in a plurality of embodiments in which a plurality of corner points of a plurality of squares 302 and 304 of the checker board serve as a plurality of feature points of the calibration target 300, the size of the plurality of squares is between adjacent feature points. Specify the interval. Thus, the plurality of image coordinates for each corner point can be determined based on its row and column positions within the checkerboard.

Once the plurality of image coordinates of the plurality of features in the image data of the object 300 are determined, then correspondence between the plurality of image coordinates and the plurality of global coordinates of the features on the object 300 can be identified. A global coordinate system may be defined for the object 300 such that multiple global coordinates for each feature correspond to the position of the feature of the object 300. For example, the origin O _w of the global coordinate system may be located at one of the four outer corners (306a, 306b, 306c, 306d) of the checkerboard, and the plane of the checkerboard may be X _w Y _{w of the} global coordinate system. Exists along a plane. Thus, in embodiments where multiple corner points serve as multiple features for calibration, the Z _w global coordinate of each corner point on the object 300 is 0, and X _w , Y of each corner point _{The w} global coordinate may be determined based on its 2D position (eg, row and column position) on the checkerboard.

  FIG. 5 shows first and second images 350, 352 of a checkerboard calibration target similar to the target 300, according to embodiments. In the embodiment of FIG. 5, the calibration object is an image displayed on the screen. However, it will be appreciated that the plurality of calibration objects described herein may be provided in other ways, such as printed on a material (eg, a piece of paper). The first image 350 and the second image 352 to be calibrated can be obtained by the left and right cameras of the binocular or stereoscopic imaging system, respectively, as part of the calibration image data set to perform camera calibration. . In particular, both images 350, 352 are to be calibrated so that all of the features to be calibrated (eg, all of the corner points of the checkerboard) are visible in both the first and second images 350, 352. (Eg, the four outer corners of the checkerboard, the four outer edges of the checkerboard, all of the black and white squares). Accordingly, the image coordinates of a plurality of features (for example, a plurality of corner points) in the first image 350, a plurality of corresponding feature image coordinates in the second image 352, and a plurality of global coordinates of the plurality of features in the calibration target. A plurality of correspondences between the left and right cameras can be identified and used to determine a plurality of calibration parameters for the left and right cameras.

  In some embodiments, if the calibration image data obtained by each camera does not capture the entire calibration object and only some of all of the features are visible in the image, Particularly when the calibration object is imaged from different camera positions and / or orientations, it may be difficult or even impossible to determine multiple image-global coordinate correspondences of multiple features. For example, referring again to FIG. 5, at least one of the first and second images 350, 352 fails to capture the entire checkerboard calibration object (eg, at least one of the four outer corners, its four Excludes at least one of the outer edges or portions, at least one of its black and white squares or portions), and only a subset of the checkerboard's corner points are visible The correspondence between the corner points and the corner points on the object can be obscured due to the repeated checkerboard pattern. This challenge can be avoided in some cases by reducing the size of the calibration object to ensure that all of the feature points are within the field of view of each camera, or by increasing the distance between the camera and the calibration object. However, these approaches can reduce the accuracy of the calibration.

  Various embodiments described herein are improved to allow determination of multiple calibration parameters of a camera even when the image data does not capture the entire calibration target and multiple calibration target features. Provide multiple calibration targets. In some embodiments, these calibration objects include one or more reference markers positioned at a plurality of defined locations. Each of the one or more fiducial markers may be uniquely identifiable in the image data to be calibrated, even if the image data depicts the calibration object at different positions and / or orientations. As a result, multiple image-global coordinate correspondences for multiple features to be calibrated can be obtained by using multiple reference markers as multiple spatial references to clearly identify each feature. Can only be determined when only visible in the image data. Advantageously, this approach allows the image to be used for camera calibration as long as the image captures all of the plurality of fiducial markers and at least some of the plurality of calibration features, thereby enabling the calibration process. Improve flexibility and robustness. This can be relatively difficult to position multiple cameras so that the entire calibration object is in the field of view of each camera, so multiple devices that use multiple cameras, such as stereoscopic vision sensing systems, can be used. It can be particularly beneficial for a UAV to implement.

  The plurality of characteristics, including at least one of shape, size, and color, of the plurality of fiducial markers described herein can be varied as desired. Exemplary reference markers include a plurality of geometric shapes (eg, points, lines, circles, triangles, quadrilaterals) symbols or letters (eg, alphanumeric characters), or a suitable combination thereof. The plurality of calibration objects provided herein may include any suitable number of fiducial markers, such as at least one, two, three, four, five, or more fiducial markers. In some embodiments, the plurality of calibration objects includes only three fiducial markers. In some embodiments, the plurality of calibration objects described herein include only a single type of fiducial marker. Alternatively, the calibration object may include different types of fiducial markers, such as at least two, three, four, five, or more different types.

  The plurality of characteristics of the plurality of fiducial markers can be selected to facilitate identification of the plurality of fiducial markers from the image data using, for example, a plurality of image recognition algorithms, feature extraction algorithms, or other computer vision techniques. Optionally, the plurality of reference markers may be visually distinct from the plurality of features to facilitate accurate identification of the plurality of reference markers and features. For example, the plurality of fiducial markers may have at least one of a shape, size, and color that is different from the plurality of features. As another example, the features may be arranged in a repeating pattern (eg, on a 1D line or a 2D surface), while the plurality of reference markers do not repeat.

  Each of the plurality of fiducial markers can be located at a different position on the calibration object. In some embodiments, the plurality of fiducial markers are in a defined arrangement so that a spatial relationship (eg, relative position, orientation) between each feature and at least one of the plurality of fiducial markers is known. Positioned relative to a plurality of features to be calibrated. For example, a plurality of fiducial markers can be overlaid with or interspersed with one or more features to be calibrated. As another example, a plurality of fiducial markers may be positioned near or adjacent to one or more features. The plurality of fiducial markers can be positioned on the calibration object such that all fiducial markers and at least some of the plurality of features are visible simultaneously in the plurality of images of the calibration object.

  Various techniques can be used to allow each fiducial marker to be uniquely identifiable in the image data to be calibrated, even if the image data represents a rotated and / or translated view of the calibration object. For example, each fiducial marker can be at least one of a different size, shape, and color, or a plurality of characteristics to allow it to be visually distinguished from other markers. Or it may contain marks. Alternatively or in combination, the plurality of fiducial markers may be arranged in a manner that allows each fiducial marker to be uniquely identifiable based on their relative arrangement. In such embodiments, the plurality of fiducial markers may be similar or identical (eg, with respect to shape, size, color, or other characteristics). For example, each fiducial marker can be positioned to form the vertices of a plurality of irregular polygons (eg, isosceles triangles, unequal triangles). At least some of the angles of the plurality of irregular polygons may have a plurality of sizes different from the other plurality of angles. Similarly, at least some of the plurality of irregular polygon edges may have a plurality of lengths different from other edges. Optionally, all of the plurality of angles can be of different sizes, and all of the plurality of edges can have different lengths. As another example, in embodiments where the features to be calibrated are arranged in a plane (eg, a 2D repeating pattern of features such as a grid), the reference markers are rotationally symmetric in the plane. A shape can be formed.

  6 and 7 illustrate a plurality of fiducial markers that are uniquely identifiable based on their relative arrangement, according to embodiments. FIG. 6 shows a first view 400 of a plurality of fiducial markers A, B, C, depicted herein as a plurality of black circles. The plurality of reference markers A, B, C can be arranged with respect to each other to form a plurality of vertices of an unequal triangle. FIG. 7 shows a second view 410 that represents rotation and translation of the first view 400. Due to the characteristics of the triangle being asymmetric, the reference markers A ′, B ′, C ′ in the second view 410 respectively correspond to the reference markers A, B, C in the first view 400. Can be uniquely identified. In some embodiments, identification of corresponding reference markers is performed based on the placement of the reference markers relative to the triangular features (eg, edges, angles) in the first and second views 400, 410. Is done. For example, in the embodiment of FIGS. 6 and 7, markers B and B ′ are both positioned on the opposite side of the longest edge of the triangle, and markers C and C ′ are both on the opposite side of the shortest edge of the triangle. Positioned. As another example, markers B and B ′ are both positioned at the vertex of the largest interior angle of the triangle, and markers C and C ′ are both positioned at the vertex of the smallest interior angle of the triangle. Although the fiducial markers in FIGS. 6 and 7 are depicted as forming a triangle, the fiducial markers can be other irregular polygons (eg, three, four, five, six, It will be appreciated that it may be arranged to form (or irregular polygons with more vertices). This approach may allow each marker to be uniquely identifiable regardless of the position and / or orientation of viewing multiple reference markers.

  FIG. 8 shows a calibration object 500 having fiducial markers 502a, 502b, 502c, according to multiple embodiments. Similar to calibration object 300, calibration object 500 includes a checkerboard having a repeating pattern of alternating black and white squares. In the embodiment of FIG. 8, the fiducial markers 502a-c are the same plurality of black circles each positioned within a different white square of the checkerboard. However, it will be appreciated that in alternative embodiments, at least some of the reference markers 502a-c may be white circles positioned within a plurality of black squares. The reference markers 502a-c are positioned at known positions with respect to the checkerboard on the calibration object 500, for example, the row and column positions of each marker are known. Each fiducial marker 502a-c may be positioned at a different portion (eg, a different row and / or column position) of the checkerboard. In some embodiments, fiducial markers 502a-c are placed on the object 500 to form a plurality of vertices of an irregular triangle. Thus, if the image of the object 500 includes all three fiducial markers 502a-c, only some of the objects 500 are visible (eg, the four outer corners of the checkerboard, the four outer edges, the multiple black Each fiducial marker 502a-c is uniquely identifiable even if it is a square and / or at least one of a plurality of white squares is excluded from the image. Once each reference marker 502a-c is identified in the image data, the spatial relationship (eg, rotation, deformation) between the plurality of reference markers in the image and the plurality of reference markers on the calibration object 500 can be determined. . This spatial relationship can then be used to determine a plurality of global coordinates for each feature (eg, corner point) depicted in the image data.

  FIG. 9 illustrates a method 600 for calibrating one or more imaging devices with a calibration object that includes one or more fiducial markers, according to embodiments. At least some of the stages of method 600 may be performed by a suitable computing system or device, such as by one or more processors. In embodiments where multiple imaging devices are mounted by a movable object, some or all of the multiple stages of method 600 may be performed when the movable object is not operating (offline calibration). Alternatively, some or all of the stages can be performed while the movable object is operating (online calibration).

  In step 610, a plurality of images to be calibrated are received from each of one or more imaging devices (eg, one or more cameras). The calibration object may include a plurality of features and one or more fiducial markers as previously described herein. The features can be arranged in any suitable configuration, for example, a repeating pattern such as a grid. The one or more fiducial markers can be designed to be uniquely identifiable in the multiple images, as described above. The plurality of images provided by each imaging device may include a plurality of images taken at different positions and / or orientations relative to the calibration object. Each image may capture all of the fiducial markers and at least some of the features to be calibrated. In some embodiments, at least one of the plurality of images generated by the plurality of imaging devices captures only a portion of all of the plurality of features to be calibrated.

  At stage 620, features and one or more reference markers in each image of the plurality of images provided by each imaging device are identified. For example, each image can be processed using various computer vision algorithms to identify multiple reference markers and multiple features visible in the image. In some embodiments, the algorithm may provide information on multiple known properties (eg, shape, size, color, relative placement) of multiple reference markers and / or features to facilitate identification. Can be executed based on. Once multiple reference markers and / or features have been identified, they can be assigned multiple image coordinates that represent their pixel locations within the image. For example, the plurality of image coordinates for the circular reference marker may be a plurality of coordinates at the center of the circle. Optionally, to improve the accuracy of the identification, for example, reduce the noise, discard multiple images that are deemed unsuitable for analysis, and so on. Can be corrected and / or pre-processed.

  10, 11, 12, and 13 illustrate the identification of features to be calibrated and reference markers in an image, according to embodiments. FIG. 10 shows an image capturing a portion of a checkerboard calibration object that includes three black circular fiducial markers, similar to the object 500. In particular, the image excludes certain parts of the calibration object, such as the upper left and lower left corners of the checkerboard, the left edge of the checkerboard, so that only some of the corner points of the checkerboard are visible. Yes. However, all three reference markers and multiple corner points near the multiple reference markers are visible. Three reference markers are placed on the calibration object to form an irregular polygon, so some areas of the calibration object are not captured by the image, but each of the multiple reference markers is uniquely identified Is possible. FIG. 11 shows the image of FIG. 10 following the process of identifying the corner points of the squares of the checkerboard. The multiple corner points may serve as multiple features for camera calibration, as previously described herein. In some embodiments, if the corner point identification algorithm fails (eg, no corner point was identified), the image is considered invalid. FIG. 12 shows the image of FIG. 10 following the process of identifying a plurality of reference markers. The plurality of circular fiducial markers can be identified using a plurality of suitable detection methods such as Hough transforms. If the algorithm fails to identify any fiducial markers, the image can be considered invalid.

  In some embodiments, the reference marker detection algorithm may generate false positives. For example, multiple regions of the image that are not reference markers can be identified. For example, the image of FIG. 12 includes several error results in which multiple portions of the image that are not fiducial markers have been misidentified. This can occur, for example, when the detection algorithm relies on shape recognition (eg, circle recognition) to identify multiple reference markers, and multiple objects of similar shape are visible in the image. FIG. 13 shows the filtering of the results of FIG. 12 to reduce the occurrence of false positives. In embodiments where a plurality of fiducial markers are interspersed with or overlaid with features to be calibrated, the image data is referenced to those within the convex hull of the identified features. It can be filtered to limit the marker identification results. In FIG. 13, the plurality of false detections outside the convex hull of the identified plurality of corner points is eliminated. A plurality of fiducial markers can be determined from the remaining results based on knowledge of the relative placement of the plurality of fiducial markers. In some embodiments, location information (eg, row and column information and / or multiple image coordinates) for each result may be determined. The location information set can then be analyzed to identify multiple reference markers that are in the correct spatial relationship with each other. For example, referring again to FIG. 13, it is known that a plurality of reference markers on the object form a plurality of vertices of a triangle, and a plurality of characteristics of the triangle (eg, edge length, angle) are As predetermined, multiple results can be searched to identify sets that will satisfy this geometric constraint. If no such result is found, or if more than one set of results is found, the image may be considered invalid.

  In step 630, a spatial relationship between one or more reference markers in each image of the plurality of images and one or more reference markers on the calibration object is determined. As described above, since the plurality of reference markers are configured to be uniquely identifiable, the positions of the plurality of reference markers in the image data are determined based on the plurality of reference markers on the actual calibration target. The spatial relationship (eg, translation, rotation, flipping) can be determined by comparing how it differs from the known positions of Such multiple differences may occur based on the position and / or orientation of the imaging device relative to the calibration object during image capture. For example, in embodiments where one or more reference markers form vertices of an irregular polygon in the image data, the spatial relationship is how the vertices of the irregular polygon in the image data are Can be determined by identifying corresponding to a plurality of known positions of a plurality of vertices of the actual object. In embodiments where a checkerboard calibration object is used and multiple reference markers are located in multiple specific squares of the checkerboard, the spatial relationship is positional information (e.g., a row) for each of the multiple reference markers in the image. And column position) and this information can be determined by comparing it with known position information (eg, row and column positions) on the actual checkerboard.

  In step 640, the correspondence between the plurality of image coordinates of the plurality of features in the image data and the plurality of global coordinates (global coordinates) for the corresponding plurality of features on the calibration object is determined in step 630. Formulated based on relationships. For example, the plurality of positions of each feature point on the actual calibration object can be determined using the determined spatial relationship. Each feature is then assigned a global coordinate set that indicates the position of the feature point relative to the global coordinate system. Thus, since the plurality of image coordinates and the plurality of global coordinates of each feature are known, a correspondence or mapping between the image and the plurality of global coordinates can be formulated.

  In embodiments where a plurality of imaging devices are calibrated and each imaging device provides a respective set of images, stage 640 identifies a plurality of features present in each of the plurality of respective image sets. And then may involve determining a plurality of global coordinates for only these features. For example, for binocular calibration using multiple checkerboard corner points, step 640 identifies multiple corner points visible in both the left and right camera images and determines multiple global coordinates for these common corner points. Can be accompanied by. Since both the left and right images depict all of the plurality of reference markers to be calibrated as described above, both the left and right images should include at least features near the plurality of reference markers. is there. In alternative embodiments, this process may be reversed and in multiple images generated by multiple imaging devices before determining which features are common across different sets of images. Multiple global coordinates can be determined for all visible features.

  In step 650, a plurality of calibration parameters for the one or more imaging devices are determined based on the correspondence formulated in step 640. As previously described herein, the plurality of calibration parameters are a plurality of parameters related to the camera internal matrix (eg, focal length, principal point location, pixel size) for each of the one or more imaging devices. ), Multiple rotation matrices and translation vectors between the camera and multiple global coordinate systems, and / or optical distortion (eg, radial distortion, eccentric distortion, thin prism distortion). Once multiple image-global coordinate correspondences for multiple features are determined, this correspondence information can be used to solve various calibration parameters, as described above with respect to step 640.

Various methods can be used to solve multiple calibration parameters. In some embodiments, initial estimates of a plurality of calibration parameters are first obtained. For example, ignoring the effects of distortion, the mapping model (eg, mapping model 200 or 250) may be used to obtain the following mapping formula between the global coordinate system and the image coordinate system for monocular camera calibration: Can be used.
Here, R = [r ₁ r ₂ r ₃ ] (3 × 3 matrix), T is a 3 × 1 vector, and λ = Z _c (Z axis of the point after being converted from the global coordinate system to the camera coordinate system) Component).

In embodiments where the calibration object is planar, eg, a planar checkerboard, the object plane is on the X _w Y _w plane of the global coordinate system such that the Z _w coordinate of each feature on the object is zero. Can be assumed to exist. Therefore, the above formula is
Can be rewritten as The homography matrix H is then
Can be defined as Here, s = 1 / λ. Λ in the homography matrix equation may correspond to an average value rather than a single point.

Since the global coordinates of the features are determined and known, for example, at step 640, the homography matrix H can be solved. Based on the definition of the homography matrix, the following formula can be obtained:
Since r ₁ and r ₂ are vectors in the rotation matrix, the following equation can be obtained:

The following can be obtained:
According to equation (8), the following can be obtained:
In Equation (10), V is a 2 × 6 matrix and is converted from the homography matrix H. To solve for b, at least three different Hs are used to constrain matrix B. Accordingly, parameter estimation can be performed using data from at least three different images of the calibration object at different positions and / or orientations.

  If matrix B can be solved, matrix M can also be solved. Subsequently, R and T can be solved according to equation (7). Based on different forms of the matrix M, the closed solution can be:

Without considering tangent effect, and
If it is,
It is.

If "tangential effect" is not considered, and
If it is,
It is.

After the initial solution is obtained, the plurality of calibration parameters can be optimized, for example, by a Levenberg-Markert algorithm. In some embodiments, the optimal solution minimizes the following objective function:

In embodiments where multiple imaging devices (eg, two or more cameras) are calibrated, stage 650 solves for the rotation matrix R and translation vector T between each of the multiple camera coordinate systems. May also be included. These external parameters can be obtained based on the multiple monocular calibration methods described herein. For example, in the binocular calibration regarding the spatial points to be observed by the left and right cameras (calibration of the two imaging devices), the coordinates P _r in the coordinate P _l and the right camera coordinate system in the left camera coordinate system satisfies the following relationship obtain.
Here, R is a rotation matrix from the right camera coordinate system to the left camera coordinate system, T is a translation vector from the origin O _l of the left camera coordinate system to the origin O _r of the right camera coordinate system.

FIG. 14 shows a model 800 of multiple spatial relationships between multiple coordinate systems in binocular camera calibration, according to embodiments. The global coordinate system (X _w Y _w Z _w with O _{w as the} origin) is related to the left camera coordinate system (X _l Y _l Z _l with O _l as the origin) by the rotation matrix R _l and the translation vector T _l. is associated to the right camera coordinate system _{(O r} was the origin _X r rZ _r) by a rotation matrix _{R r} and translation vector _{T r.} These rotation matrices and translation vectors can be obtained using a number of methods previously described herein. The rotation matrix R and translation vector T for converting between the left and right camera coordinate systems can then be determined by the following relationship:

  Once a plurality of calibration parameters are determined according to a plurality of methods described herein, the parameters may be used to generate 3D spatial information for various downstream applications, such as generating multiple maps. Can be used to process image data obtained by multiple imaging devices to be suitable for image restoration, object recognition, navigation, and the like. For example, image data is processed based on a plurality of determined calibration parameters to correct for multiple effects of optical distortion associated with one or more imaging devices, also referred to herein as “distortion correction”. obtain. As another example, in embodiments where two or more imaging devices are used to capture a plurality of respective image data sets, for example, the image data set may be stereoscopic. Multiple calibration parameters can be used to allow them to be used.

FIG. 15 illustrates a model 900 for mapping between a plurality of distortion image coordinates and a plurality of distortion correction image coordinates, according to embodiments. Distortion can occur in multiple images captured by the imaging device due to multiple optical characteristics of the lens. The image distortion correction is performed using a plurality of distortion correction images or pixel coordinates (u ₂ , corresponding to a plurality of distortion correction image plane coordinates (X _p , Y _p ) according to an ideal pinhole camera model. v ₂₎ so as to obtain an image) having the captured image by the imaging device (e.g., a plurality of distorted image plane coordinates (X _d, a plurality of strain images or pixel coordinates corresponding to _{Y d) (u 1, v} 1 Can be used to remove multiple effects of distortion in the image. Accordingly, image distortion correction can be used to convert a distortion image into a distortion correction image. The image distortion correction procedure may be performed based on a plurality of previously determined internal parameters (eg, distortion parameters, camera internal matrix M) for a plurality of imaging devices. Such multiple parameters may be determined using multiple approaches presented herein.

Various methods can be used to perform image distortion correction. For example, a plurality of distortion-corrected image plane coordinates (X _p , Y _p ) can be determined from a plurality of pixel coordinates (u ₂ , v ₂ ) that have been distortion-corrected according to the following equation.
The plurality of distortion image plane coordinates (X _d , Y _d ) can be represented by a mathematical distortion model (eg, the model described by the distortion equation (3) presented earlier herein) and a plurality of distortions. Based on a plurality of previously determined values for the parameter, it can be determined from the distortion corrected image plane coordinates (X _p , Y _p ). The plurality of pixel coordinates of the distorted image (u ₁ , v ₁ ) can then be obtained by the following formula:
The gray scale values of the plurality of distortion corrected pixel coordinates (u ₂ , v ₂ ) can be obtained by performing interpolation on the plurality of distortion pixel coordinates (u ₁ , v ₁ ). Interpolation may be performed using any suitable method such as nearest neighbor interpolation or bilinear interpolation. Therefore, a distortion corrected image can be obtained.

  Similar to the image distortion correction processing described herein, image correction, also known as stereo correction, can involve image-to-image conversion processing. However, image correction also involves spatial transformation at the camera coordinate system level. Image modification processes image data captured by multiple imaging devices to facilitate implementation of a stereoscopic algorithm, for example, for generation of depth maps, parallax maps, and other 3D representations of a scene. Can be used. For example, in some embodiments, multiple stereoscopic algorithms may involve determining multiple point correspondences between multiple images captured by different multiple cameras (eg, left camera and right camera). Image correction can be used to project the image data obtained by each camera onto a common plane, thereby simplifying the procedure of searching for multiple matching image points in multiple images.

16 and 17 illustrate image correction procedures for a binocular camera system, according to embodiments. FIG. 16 shows a model 1000 of mapping between the left camera coordinate system (camera 1) and the right camera coordinate system (camera 2) for stereoscopic viewing. The image modification process may _involve determining rotation matrices R ₁ , R _r , and R _rect that are used to modify the left and right image data. In some embodiments, these matrices are configured to maximize multiple overlapping portions of the left and right images after correction and minimize the amount of image deformation. Furthermore, if the same matrix M is used in both modified images, the modified images can be at the same scale, thereby facilitating depth of field calculations.

FIG. 17 shows a model 1050 for mapping between an unmodified camera coordinate system and a modified camera coordinate system. In some embodiments, the rotation matrices R _l and R _r are
Can meet. Here, R is a rotation matrix from the right camera coordinate system (X _r , Y _r , Z _r ) to the left camera coordinate system (X _l , Y _l , Z _l ). R _l and R _r are respectively the first temporary coordinate system from the left camera coordinate system.
To and from the right camera coordinate system to a second temporary coordinate system
Is a multiple rotation matrix. In some embodiments,
May be set equal to R _r such that the rotation angles of the rotation matrix R are equally divided. The left and right camera coordinate systems can be transformed according to R ₁ and R _r , respectively, so that the spatial poses of the two camera coordinate systems are the same after transformation. However, there may be a translation vector T _temp = R ₁ T in the coordinate system obtained by transforming the left camera coordinate system according to R ₁ .

The two coordinate systems can then be transformed simultaneously by the rotation matrix R _rect so that the XOY planes of the two coordinate systems are coplanar and both X axes are parallel to the reference line. The matrix R _rect can be determined subjectively. For example, R _rect is
It can be expressed as here,
And
, And _the a _{e 3 = e 1 × e 2} , a _{T temp = R l T (3} × 1 _{matrix), T} temp_x _is defined as the first row and first column of the _{T temp, T temp_y} is , T _temp is defined as the first column of the second row.

In some embodiments, after rotation of multiple camera coordinate systems, there is only a translation vector T _rect = R _rect R _l T that is parallel to the X axis and between the two coordinate systems. Optionally, matrix M can be replaced with matrix M ₁ (see FIG. 16).

After the rotation matrices R _l , R _r , and R _rect and the mapping matrix M are determined, an image correction procedure can be performed similar to the single image distortion correction procedure described above. Image correction can be performed separately for each set of images obtained by each individual imaging device. For example, referring again to FIG. 16, for point (u ₃ , v ₃ ), an inverse mapping can be performed to obtain the coordinates (X _pl , Y _pl , Z _pl ) in the original left camera coordinate system.
This can be scaled to obtain the following relationship:

The plurality of strain coordinates may be obtained according to a plurality of strain parameters and a strain model (eg, Equation (3)) determined previously. A plurality of pixel coordinates (u ₁ , v ₁ ) in the original left coordinate system can then be obtained.

The grayscale values of the modified plurality of image coordinates (u ₃ , v ₃ ) can be determined using any suitable interpolation algorithm. This modification process can also be used to obtain a plurality of modified image coordinates for the right camera.

  FIG. 18 shows a plurality of binocular camera images after a process of performing image correction, according to embodiments. The plurality of images include a first modified image 1100 obtained by the left camera and a second modified image 1102 obtained by the right camera. In particular, the objects in the first image 1100 are horizontally aligned with the corresponding objects in the second image 1102 as indicated by the superimposed horizontal lines 1104.

  Once the image data obtained from one or more images has been processed with distortion correction and / or correction as described herein, the multiple images may be used for a variety of applications, such as multiple computer vision applications. Can be used. In some embodiments, multiple images may be used to recover spatial information regarding the imaged scene, such as multiple distances from multiple imaging devices of multiple objects in the scene. This spatial information can be used to generate multiple 3D representations of the environment, such as depth maps, parallax maps, occupancy grids, point clouds, and others.

  FIG. 19 illustrates a method 1200 for generating multiple depth maps from image data, according to multiple embodiments. At least some of the stages of method 1200 may be performed by a suitable computing system or device, such as by one or more processors.

  In step 1210, one or more images are obtained from each of the one or more imaging devices. The imaging devices may be configured to capture multiple images at the same time, or substantially simultaneously, to generate corresponding multiple images of the scene. The imaging devices can be positioned with respect to each other to produce multiple images of the same scene from different multiple locations and / or orientations. For example, a UAV equipped with a binocular vision system may have a left camera and a right camera that capture multiple images of the environment surrounding the UAV simultaneously, but from two different positions and / or orientations. Thus, multiple objects in a scene will have multiple image coordinates that are different in multiple images obtained by different multiple imaging devices.

  In step 1220, one or more images from each of the one or more imaging devices are processed based on a plurality of calibration parameters for the one or more imaging devices to generate a 3D depth map. The plurality of calibration parameters may include a plurality of internal and / or external parameters for a plurality of imaging devices, as previously described herein, and a plurality of calibration methods described herein, such as method 600. Can be determined in advance. The calibration method can be an online calibration method or an offline calibration method. In some embodiments, the plurality of calibration parameters is determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices. The calibration object can be any of the embodiments presented herein, for example, a checkerboard calibration object. For example, the calibration object may include a plurality of features arranged in a repeating pattern and one or more reference markers. One or more fiducial markers are described above, for example, with respect to the embodiments of FIGS. 6, 7 and 8, such that they can be uniquely identified within each calibration image obtained by a plurality of imaging devices. Each can be designed as such. In some embodiments, at least one of the plurality of calibration images captures only a portion of all of the plurality of features to be calibrated. As previously described, the methods described herein are such that some or all of the images capture all features of the feature to be calibrated as described herein. Even when not, it can be used to determine a plurality of calibration parameters from a plurality of calibration images.

  Various methods of processing the images can be used so that a 3D depth map can be generated from multiple images. For example, the plurality of images may be processed by the image distortion correction and / or correction techniques described herein. The modified images can then be used as input to various stereoscopic algorithms to restore depth information about the objects in the images and thus generate a depth map of the imaged scene. For example, multiple images obtained by different imaging devices (eg, multiple images captured simultaneously or substantially simultaneously) can be compared to one another to determine multiple parallaxes between the images. Based on the disparity information and information regarding the spatial relationship between the plurality of imaging devices, a distance between the object and the plurality of imaging devices may be determined. For example, the parallax between multiple image coordinates of the same object in multiple images from different imaging devices can be calculated. The distance from the plurality of imaging devices to the object may be expressed in terms of parallax, focal lengths of the plurality of imaging devices, and spatial relationships between the plurality of imaging devices (eg, distances between the plurality of imaging devices). Thus, once the focal length and spatial relationship information are determined (eg, via a plurality of calibration methods described herein), the distance can be calculated.

  This process can be repeated to generate a 3D depth map for all points in the imaged scene. Such multiple depth maps can be used for a wide variety of applications. For example, in embodiments where multiple imaging devices are mounted by a UAV, a depth map generated based on image data from multiple imaging devices provides information indicating multiple distances from various objects to the UAV. Can be used to provide. This information can be used to facilitate various UAV functions such as obstacle avoidance, navigation, object tracking, and the like.

  The multiple systems, devices, and methods described herein can be applied to a wide variety of movable objects. As previously mentioned, any description of an aircraft herein may apply to and be used for any movable object. The movable object of the present invention can be in the air (eg, fixed wing aircraft, rotary wing aircraft, or aircraft having neither fixed wing nor rotary wing), underwater (eg, ship or submarine), ground (eg, car, truck, bus). , Cars such as vans, motorcycles, movable structures or frames such as rods, fishing rods, or trains), underground (eg, subways), outer space (eg, space planes, satellites, or space probes), or these environments It can be configured to move in any suitable environment, such as any combination. The movable object may be a transporter, such as the transporter described elsewhere herein. In some embodiments, the movable object may be mounted on a living body such as a human or animal. Suitable animals may include birds, canines, felines, equines, bovine animals, sheep, pigs, dolphins, rodents, or insects.

  The movable object may be able to move freely in the environment with respect to 6 degrees of freedom (eg, translational 3 degrees of freedom and rotational 3 degrees of freedom). Alternatively, the movement of the movable object may be limited with respect to one or more degrees of freedom, such as by a given path, trajectory, or orientation. The movement can be actuated by any suitable actuation mechanism such as an engine or a motor. The actuation mechanism of the movable object can be powered by any suitable energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. The movable object may self-propell through the propulsion system, as described elsewhere herein. Optionally, the propulsion system may operate with an energy source such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, movable objects can be carried by creatures.

  In some cases, the movable object may be a transporter. Suitable multiple vehicles may include water vehicles, aircraft, space vehicles, or ground vehicles. For example, a plurality of aircraft may be a fixed wing aircraft (eg, airplane, glider), a rotary wing aircraft (eg, helicopter, rotorcraft), an aircraft having both fixed and rotary wings, or an aircraft that does not have any (eg, Small soft airship, hot air balloon). A transporter may be self-propelled, such as self-propelled in the air, water, underwater, outer space, ground, or underground. A self-propelled vehicle may utilize a propulsion system such as a propulsion system that includes one or more engines, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles, or any suitable combination thereof. . In some cases, a movable object can take off from the surface, land on the surface, maintain its current position and / or orientation (eg, hover), change orientation, and / or change position A propulsion system can be used to make it happen.

  The movable object can be controlled remotely by the user or can be controlled closer to the occupant in or on the movable object. In some embodiments, the movable object may be an unmanned movable object such as a UAV. An unmanned movable object such as a UAV may not have a passenger on the movable object. The movable object may be controlled by a person or an autonomous control system (eg, a computer control system) or by any suitable combination of these. The movable object can be an autonomous or semi-autonomous robot such as a robot configured using artificial intelligence.

  The movable object may have any suitable size and / or multiple dimensions. In some embodiments, the movable object may be of a size and / or multiple dimensions for having an occupant who is a person in or on the transport. Alternatively, the movable object may be smaller than the size and / or dimensions capable of having an occupant who is a person in or on the transport. The movable object may be of a size and / or multiple dimensions suitable for being lifted or mounted by a person. Alternatively, the movable object may be larger than a size and / or multiple dimensions suitable for being lifted or mounted by a person. In some cases, the movable object has a maximum dimension (eg, length, width, height, diameter, diagonal, less than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m). ). The maximum dimension may be greater than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance between the shafts of the opposing rotors of the movable object may be less than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. . Alternatively, the distance between the shafts of the opposing rotors may be greater than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.

In some embodiments, the movable object may have a volume of less than 100 cm × 100 cm × 100 cm, less than 50 cm × 50 cm × 30 cm, or less than 5 cm × 5 cm × 3 cm. The total volume of the movable object is approximately 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , ^{200cm 3, 300cm 3, 500cm 3} , 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, 100,000cm 3, 1m 3 or may be smaller than 10 m ^3,, or may be equal to them. Conversely, the total volume of the movable object is approximately 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , ^{150cm 3, 200cm 3, 300cm 3} , 500cm 3, 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, 100,000cm 3, 1m 3 or may be greater than 10 m ^3,, or may be equal to those .

In some embodiments, the movable object is approximately ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or 5 cm ² less than Or they may have an equal footprint (which may refer to the transverse cross-sectional area surrounded by the movable object). Conversely, the footprint is approximately ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or may be greater than 5 cm ^2, or They may be equal.

  In some cases, the movable object may have a weight of 1000 kg or less. The weight of the movable object is approximately 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg. , 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg, or may be equal. Conversely, the weight is approximately 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg. , 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg may be heavier or equal.

  In some embodiments, the movable object may be small relative to the load carried by the movable object. The load may include a load and / or support mechanism as described in more detail below. In some examples, the ratio of the weight of the movable object to the load may be greater than, less than, or equal to approximately 1: 1. In some cases, the ratio of the weight of the movable object to the load may be greater than, less than, or equal to approximately 1: 1. Optionally, the ratio of the weight of the support mechanism to the load may be greater than, less than, or equal to approximately 1: 1. When desired, the ratio of the weight of the movable object to the load may be less than or equal to 1: 2, 1: 3, 1: 4, 1: 5, 1:10, or even It may be small. Conversely, the ratio of the weight of the movable object to the load may be greater than or equal to 2: 1, 3: 1, 4: 1, 5: 1, 10: 1, or even greater. Also good.

  In some embodiments, the movable object may consume a small amount of energy. For example, the movable object may use less than about 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or even less. In some cases, the support mechanism for the movable object may consume a small amount of energy. For example, the support mechanism may use less than approximately 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or even less. Optionally, the load of movable objects may consume a small amount of energy, such as approximately 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than 1 W / h, or even less.

  FIG. 20 illustrates an unmanned aerial vehicle (UAV) 1300 according to embodiments of the present invention. A UAV may be an example of a movable object as described herein. The UAV 1300 may include a propulsion system having four rotors 1302, 1304, 1306, and 1308. Any number of rotors (eg, 1, 2, 3, 4, 5, 6, or more) may be provided. The multiple rotors can be multiple embodiments of multiple self-tightening rotors described elsewhere herein. Multiple rotors, rotor assemblies, or other propulsion systems of an unmanned aerial vehicle may allow the unmanned aircraft to hover / maintain position, change orientation, and / or change position. The distance between the shafts of the opposing rotors can be any suitable length 1310. For example, the length 1310 may be less than or equal to 2 m, or may be less than or equal to 5 m. In some embodiments, the length 1310 can be in the range of 40 cm to 1 m, 10 cm to 2 m, or 5 cm to 5 m. Any description of UAV herein may apply to movable objects, such as different types of movable objects, and vice versa.

  In some embodiments, the movable object may be configured to carry a load. A cargo may include one or more of passengers, cargo, equipment, equipment, and the like. A load may be provided in the housing. The housing may be separate from the housing of the movable object and may be part of the housing for the movable object. Alternatively, the load may be provided with a housing, while the movable object does not have a housing. Alternatively, multiple portions of the load, or the entire load may be provided without a housing. The load can be firmly fixed to the movable object. Optionally, the load can be movable relative to the movable object (eg, translatable or rotatable relative to the movable object).

  In some embodiments, the load includes a load. The load may be configured not to perform any operation or function. Alternatively, the load may be a load configured to perform an action or function also known as a functional load. For example, a load may include one or more sensors for investigating one or more objects. Any suitable sensor such as an image capture device (eg, camera), audio capture device (eg, parabolic microphone), infrared imaging device, or ultraviolet imaging device may be incorporated into the load. The sensor may provide static sensing data (eg, photographs) or dynamic sensing data (eg, video). In some embodiments, the sensor provides sensing data regarding the object of the load. Alternatively or in combination, the load may include one or more transmitters for providing multiple signals to one or more objects. Any suitable transmitter such as an illumination light source or a sound source can be used. In some embodiments, the load includes one or more transceivers, such as for communication with a module that is remote from the movable object. Optionally, the load can be configured to interact with the environment or object. For example, a load may include tools, instruments, or mechanisms such as a robot arm that can manipulate multiple objects.

  Optionally, the load can include a support mechanism. A support mechanism may be provided for the load, and the load may be directly (eg, in direct contact with the movable object) or indirectly (eg, not in contact with the movable object) via the support mechanism. And can be connected to a movable object. Conversely, the load can be mounted on a movable object without requiring a support mechanism. The load can be formed integrally with the support mechanism. Alternatively, the load can be releasably connected to the support mechanism. In some embodiments, the load may include one or more load elements, where one or more of the plurality of load elements is relative to the movable object and / or support mechanism as described above. It may be possible to move.

  The support mechanism can be formed integrally with the movable object. Alternatively, the support mechanism can be releasably connected to the movable object. The support mechanism may be directly or indirectly connected to the movable object. The support mechanism may provide support for the load (eg, carry at least a portion of the weight of the load). The support mechanism can include a suitable mounting structure (eg, a gimbal platform) that can stabilize and / or direct the movement of the load. In some embodiments, the support mechanism can be adapted to control the state (eg, position and / or orientation) of the load relative to the movable object. For example, the support mechanism may be relative to the movable object (eg, 1, 2, or 3) such that the load maintains its position and / or orientation relative to a suitable reference frame regardless of the movement of the movable object. 2 degrees of translation and / or 1, 2 or 3 degrees of rotation). The reference frame may be a fixed reference frame (eg, the ambient environment). Alternatively, the reference frame may be a moving reference frame (eg, movable object, load object).

  In some embodiments, the support mechanism may be configured to allow movement of the load relative to the support mechanism and / or the movable object. Movement can be translation with up to 3 degrees of freedom (eg, along 1, 2 or 3 axes), rotation with up to 3 degrees of freedom (eg, around 1, 2 or 3 axes), or any of these Any suitable combination.

  In some cases, the support mechanism may include a support mechanism frame assembly and a support mechanism actuation assembly. The support mechanism frame assembly may provide structural support for the load. The support mechanism frame assembly may include a plurality of individual support mechanism frame parts, some of which may be movable relative to each other. The support mechanism actuation assembly may include one or more actuators (eg, motors) that actuate movement of a plurality of individual support mechanism frame components. The plurality of actuators may allow movement of a plurality of support mechanism frame parts simultaneously, or may be configured to allow movement of a single support mechanism frame part at a time. The movement of the plurality of support mechanism frame parts can cause a corresponding movement of the load. For example, the support mechanism actuation assembly may actuate rotation of one or more support mechanism frame components around one or more rotation axes (eg, roll axis, pitch axis, or yaw axis). The rotation of the one or more support mechanism frame components may cause the load to rotate about one or more axes of rotation relative to the movable object. Alternatively or in combination, the support mechanism actuation assembly may actuate translation of one or more support mechanism frame components along one or more translation axes, thereby providing one or more to the movable object. Can cause translation of the load along the corresponding axis.

  In some embodiments, the movement of movable objects, support mechanisms, and loads relative to a fixed reference frame (eg, the ambient environment) and / or relative to each other can be controlled by the terminal. A terminal may be a remote control device at a distance from a movable object, a support mechanism, and / or a load. The terminal can be placed on or affixed to the support platform. Alternatively, the terminal can be a handheld device or a wearable device. For example, the terminal may include a smartphone, tablet, laptop, computer, glasses, gloves, helmet, microphone, or any suitable combination thereof. The terminal may include a user interface such as a keyboard, mouse, joystick, touch screen, or display. Any suitable user input such as manually entered commands, voice control, gesture control, position control (eg, via terminal movement, position, or tilt) may be used to interact with the terminal.

  The terminal can be used to control any suitable state of the movable object, the support mechanism, and / or the load. For example, the terminals can be used to control the position and / or orientation of movable objects, support mechanisms, and / or loads relative to and / or relative to each other relative to a fixed reference. In some embodiments, the terminal controls a moving mechanism such as a support mechanism actuation assembly, a load sensor, a load transmitter, a support mechanism, and / or a plurality of individual elements of the load. Can be used. A terminal may include a wireless communication device adapted to communicate with one or more of a movable object, a support mechanism, or a load.

  The terminal may include a display unit suitable for viewing information on movable objects, support mechanisms, and / or loads. For example, the terminal is configured to display movable object, support mechanism, and / or load information regarding position, translation speed, translation acceleration, orientation, angular velocity, angular acceleration, or any suitable combination thereof. Can be done. In some embodiments, the terminal may display information provided by the load, such as data provided by the functional load (eg, multiple images recorded by a camera or other image capture device). .

  Optionally, the same terminal can control the state of the movable object, support mechanism, and / or load, or the state of the moveable object, support mechanism, and / or load, and from the movable object, support mechanism, and / or load. Both receiving and / or displaying information can be performed. For example, the terminal may control the placement of the load relative to the environment while displaying image data captured by the load or information about the position of the load. Alternatively, different terminals may be used for different functions. For example, the first terminal may control the movement or state of the movable object, the support mechanism, and / or the load, while the second terminal receives and / or receives information from the movable object, the support mechanism, and / or the load. Or you can display. For example, the first terminal can be used to control the placement of the load relative to the environment, while the second terminal can display the image data captured by the load. Between a movable object and an integrated terminal that controls both the movable object and receives data, or between a movable object and multiple terminals that both control the movable object and receive data Various communication modes can be used. For example, at least two different communication modes may be formed between a movable object and a terminal that both controls the movable object and receives data from the movable object.

  FIG. 21 illustrates a movable object 1400 including a support mechanism 1402 and a load 1404, according to embodiments. Although the movable object 1400 is depicted as an aircraft, this depiction is not intended to be limiting, and any suitable type of movable object may be used as previously described herein. Those skilled in the art will appreciate that any of the embodiments described herein in the context of multiple aircraft systems may be applied to any suitable movable object (eg, UAV). In some cases, the load 1404 may be provided on the movable object 1400 without requiring a support mechanism 1402. The movable object 1400 can include a plurality of propulsion mechanisms 1406, a sensing system 1408, and a communication system 1410.

  The multiple propulsion mechanisms 1406 include multiple rotors, multiple propellers, multiple blades, multiple engines, multiple motors, multiple wheels, multiple axles, multiple magnets, multiple nozzles, as previously described. One or more may be included. For example, the plurality of propulsion mechanisms 1406 may be a plurality of self-tightening rotors, a plurality of rotor assemblies, or other plurality of rotary propulsion units, as disclosed elsewhere herein. . The movable object may have one or more, two or more, three or more, four or more propulsion mechanisms. The plurality of propulsion mechanisms can all be of the same type. Alternatively, the one or more propulsion mechanisms may be different types of propulsion mechanisms. The plurality of propulsion mechanisms 1406 may be mounted on the movable object 1400 using any suitable means such as a support element (eg, a drive shaft) as described elsewhere herein. The plurality of propulsion mechanisms 1406 may be mounted on any suitable portion of the movable object 1400, such as the top, bottom, front, rear, sides, or any suitable combination of these.

  In some embodiments, the plurality of propulsion mechanisms 1406 allow the movable object 1400 to take off vertically from the surface without requiring any horizontal movement of the movable object 1400 (eg, without sliding on the runway). Or it may be possible to land vertically on the surface. Optionally, the plurality of propulsion mechanisms 1406 may be operable to allow the movable object 1400 to hover in the air at a specified position and / or orientation. One or more of the plurality of propulsion mechanisms 1400 may be controlled independently of other propulsion mechanisms. Alternatively, multiple propulsion mechanisms 1400 can be configured to be controlled simultaneously. For example, the movable object 1400 may have a plurality of horizontal plane oriented rotors that may provide lift and / or thrust to the movable object. The plurality of horizontal plane oriented rotors may be operated to provide vertical takeoff, vertical landing, and hovering capabilities to the movable object 1400. In some embodiments, one or more of the plurality of horizontal plane oriented rotors may rotate in a clockwise direction while one or more of the plurality of horizontal plane oriented rotors may rotate in a counterclockwise direction. For example, the number of clockwise rotors may be equal to the number of counterclockwise rotors. The rotational speed of each of the plurality of horizontally oriented rotors controls the lift and / or thrust generated by each rotor, and thereby (for example, for up to 3 degrees of translation and up to 3 degrees of rotation). It can be varied independently to adjust the spatial arrangement, velocity, and / or acceleration of the movable object 1400.

  Sensing system 1408 may include one or more sensors that can sense the spatial arrangement, velocity, and / or acceleration of movable object 1400 (eg, with respect to up to three degrees of translation and up to three degrees of rotation). The one or more sensors may include a global positioning system (GPS) sensor, a motion sensor, an inertial sensor, a proximity sensor, or an image sensor. Sensing data provided by the sensing system 1408 may indicate the spatial arrangement, velocity, and / or orientation of the movable object 1400 (eg, using a suitable processing unit and / or control module, as described below). Can be used to control. Alternatively, the sensing system 1408 may be used for environments surrounding moving objects, such as weather conditions, proximity to potential obstacles, locations of multiple geographic features, locations of multiple artificial structures, and the like. Can be used to provide data.

  The communication system 1410 enables communication with a terminal 1412 having the communication system 1414 via a plurality of radio signals 1416. The communications system 1410, 1414 may include any number of transmitters, receivers, and / or transceivers suitable for wireless communications. The communication can be a one-way communication so that data can only be transmitted in one direction. For example, one-way communication may involve only the movable object 1400 transmitting data to the terminal 1412, or vice versa. Data may be transmitted from one or more transmitters of communication system 1410 to one or more receivers of communication system 1412, or vice versa. Alternatively, the communication can be bi-directional communication so that data can be transmitted between the movable object 1400 and the terminal 1412 in both directions. Two-way communication may involve transmitting data from one or more transmitters of communication system 1410 to one or more receivers of communication system 1414, and vice versa.

  In some embodiments, the terminal 1412 may provide control data for one or more of the movable object 1400, the support mechanism 1402, and the load 1404, and the mobile object 1400, the support mechanism 1402, and the load 1404. Information (eg, data sensed by the load, such as position and / or movement information of a movable object, support mechanism or load, image data captured by the load camera, etc.) may be received from one or more. In some cases, control data from the terminal may include multiple commands for multiple relative positions, multiple movements, multiple actuations, or multiple controls of the movable object, support mechanism, and / or load. Can be included. For example, the control data may cause the position and / or orientation of the movable object to be changed (eg, via control of a plurality of propulsion mechanisms 1406), or the load (eg, a support mechanism) relative to the movable object. (Via 1402 control). Control data from the terminal to control the operation of the camera or other image capture device (eg, still or video capture, zoom in or out, power on / off, switch between multiple imaging modes, change image resolution, focus , Change of depth of field, change of exposure time, change of viewing angle or field of view) and the like can be controlled. In some cases, the plurality of communications from the movable object, the support mechanism, and / or the load may include information from one or more sensors (eg, of the sensing system 1408 or of the load 1404). The plurality of communications may include sensed information from one or more different types of sensors (eg, GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors). Such information may relate to the position (eg, position, orientation), movement, or acceleration of movable objects, support mechanisms, and / or loads. Such information from the load may include data captured by the load or a sensed state of the load. Control data transmitted and provided by the terminal 1412 may be configured to control one or more states of the movable object 1400, the support mechanism 1402, or the load 1404. Alternatively, or in combination, the support mechanism 1402 and the load 1404 are each such that the terminal 1412 can communicate independently with each of the movable object 1400, the support mechanism 1402, and the load 1404, respectively. A communication module configured to communicate with the terminal for control can also be included.

  In some embodiments, the movable object 1400 may be configured to communicate with other remote devices in addition to or in place of the terminal 1412. The terminal 1412 can also be configured to communicate not only with the movable object 1400 but also with other remote devices. For example, movable object 1400 and / or terminal 1412 may communicate with other movable objects, or other movable object support mechanisms or loads. When desired, the remote device can be a second terminal or other computing device (eg, a computer, laptop, tablet, smartphone, or other mobile device). The remote device may be configured to transmit data to the movable object 1400, receive data from the movable object 1400, transmit data to the terminal 1412, and / or receive data from the terminal 1412. Optionally, the remote device may be connected to the Internet or other telecommunications network so that data received from movable object 1400 and / or terminal 1412 can be uploaded to a website or server.

  FIG. 22 is a schematic diagram using a block diagram of a system 1500 for controlling a movable object, according to embodiments. System 1500 can be used in combination with any suitable embodiment of multiple systems, devices, and methods disclosed herein. System 1500 can include a sensing module 1502, a processing unit 1504, a non-transitory computer readable medium 1506, a control module 1508, and a communication module 1510.

  Sensing module 1502 may utilize different types of sensors that collect information related to multiple movable objects in different ways. Different types of sensors may sense different types of signals or signals from different sources. For example, the plurality of sensors may include an inertial sensor, a GPS sensor, a proximity sensor (eg, a rider), or a visual / image sensor (eg, a camera). The sensing module 1502 can be operatively connected to a processing unit 1504 having a plurality of processors. In some embodiments, the sensing module is operable with a transmission module 1512 (eg, a Wi-Fi® image transmission module) configured to transmit the sensing data directly to a suitable external device or system. Can be connected to. For example, the transmission module 1512 can be used to transmit multiple images captured by the camera of the sensing module 1502 to a remote terminal.

  The processing unit 1504 may include one or more processors such as a programmable processor (eg, a central processing unit (CPU). The processing unit 1504 may be operatively connected to a non-transitory computer readable medium 1506. Non-transitory. The computer readable medium 1506 may store logic, code, and / or a plurality of program instructions that can be executed by the processing unit 1504 to perform one or more steps. Or may include multiple memory units (eg, removable media such as an SD card or random access memory (RAM) or external storage) In some embodiments, data from the sensing module 1502 is non-temporary The information may be transmitted directly to and stored in the plurality of memory units of computer readable medium 1506. The plurality of memory units of non-transitory computer readable medium 1506 may be any of the methods described herein. May store logic, code, and / or multiple program instructions executable by the processing unit 1504. For example, the processing unit 1504 may include one or more processors in the processing unit 1504, A plurality of instructions may be configured to execute the sensing data generated by the sensing module, and the plurality of memory units may store the sensing data from the sensing module to be processed by the processing unit 1504. Real In embodiment, the plurality of memory units of non-transitory computer-readable medium 1506 may be used to store a plurality of processing results generated by the processing unit 1504.

  In some embodiments, the processing unit 1504 may be operably connected to a control module 1508 that is configured to control the state of the movable object. For example, the control module 1508 can be configured to control multiple propulsion mechanisms of the movable object to adjust the spatial arrangement, speed, and / or acceleration of the movable object with respect to six degrees of freedom. Alternatively or in combination, the control module 1508 may control one or more of the state of the support mechanism, load, or sensing module.

  The processing unit 1504 is operable to a communication module 1510 configured to send data to and / or receive data from one or more external devices (eg, terminal, display device, or other remote controller). Can be connected to. Any suitable communication means such as wired communication or wireless communication can be used. For example, the communication module 1510 can be a local area network (LAN), wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network, telecommunications network, cloud communication, and the like One or more of the things may be utilized. Optionally, multiple relay stations such as towers, satellites, mobile stations may be used. Multiple wireless communications may be proximity dependent or proximity independent. In some embodiments, a line of sight may or may not be required for multiple communications. The communication module 1510 receives one or more of sensing data from the sensing module 1502, multiple processing results generated by the processing unit 1504, given control data, multiple user commands from the terminal or remote controller, and the like. They can transmit and / or receive them.

  The multiple components of system 1500 may be arranged in any suitable configuration. For example, one or more of the components of system 1500 are located on a movable object, support mechanism, load, terminal, sensing system, or additional external device in communication with one or more of the foregoing. obtain. In addition, while FIG. 22 depicts a single processing unit 1504 and a single non-transitory computer readable medium 1506, this is not intended to be limiting and that the system 1500 may have multiple processing. Those skilled in the art will appreciate that a unit and / or non-transitory computer readable medium may be included. In some embodiments, one or more of the plurality of processing units and / or non-transitory computer readable media may be any suitable aspect of processing and / or memory functions performed by the system 1500, As may occur at one or more of different locations, such as on a mechanism, load, terminal, sensing module, additional external device in communication with one or more of the above, or a suitable combination of these, It can be positioned at a plurality of positions described above.

  As used herein, A and / or B includes one or more of A or B, and combinations of these, such as A and B.

  The specific dimensions of any of the apparatus, devices, systems, and components thereof of the present disclosure are intended for use as would be apparent to one of ordinary skill in the art in view of the present disclosure. It can be easily changed according to. Furthermore, the embodiments described herein are for illustrative purposes only, and various modifications or changes in view of them may be suggested to those skilled in the art, and the spirit and scope of the present application and attached It is understood that this is included in the aspects of the following claims. Many different combinations of the embodiments described herein are possible, and such combinations are considered part of this disclosure. In addition, all features described in the context of any one embodiment herein can be readily adapted for use in other embodiments herein.

While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will envision many variations, modifications, and substitutions without departing from the invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention. The following claims define the scope of the invention, and it is intended that the claims cover the methods and structures within the scope of these claims, and their equivalents.
(Item 1)
A method for calibrating one or more imaging devices,
Receiving, from each of the one or more imaging devices, a plurality of images to be calibrated comprising a plurality of features arranged in a repeating pattern and one or more reference markers;
Determining a spatial relationship between the one or more reference markers in each of the plurality of images and the one or more reference markers to be calibrated with the assistance of one or more processors. When,
Based on the spatial relationship and with the assistance of one or more processors, image coordinates of a plurality of features in each image of the plurality of images, and a plurality of global coordinates for a plurality of corresponding features on the calibration object Formulating the correspondence between
Determining a plurality of calibration parameters for the one or more imaging devices based on the correspondence and with the assistance of the one or more processors;
Each of the one or more reference markers is uniquely identifiable in each of the plurality of images,
The method wherein the plurality of images includes at least one image that captures only some of all of the plurality of features.
(Item 2)
The method of item 1, wherein the one or more imaging devices are mounted by an unmanned aerial vehicle.
(Item 3)
2. The method of item 1, wherein each of the plurality of images captures the one or more reference markers and at least a subset of the plurality of features to be calibrated.
(Item 4)
The method of item 1, wherein the plurality of images from each of the one or more imaging devices are imaged at different positions and orientations relative to the calibration object.
(Item 5)
2. The method according to item 1, wherein the calibration target is a checkerboard including a plurality of squares, and the plurality of features include a plurality of corner points of the plurality of squares.
(Item 6)
6. The method of item 5, wherein each of the one or more reference markers is positioned in one of the plurality of squares.
(Item 7)
6. The method of item 5, wherein the plurality of features includes four outer corners of the checkerboard and the at least one image excludes at least one of the four outer corners.
(Item 8)
The method of item 1, wherein the one or more reference markers comprises at least three reference markers.
(Item 9)
The method of item 1, wherein the one or more reference markers comprises a plurality of circular dots, triangles, or quadrilaterals.
(Item 10)
The method of item 1, wherein the one or more reference markers are overlaid on the repeating pattern.
(Item 11)
The method of item 1, wherein the one or more reference markers form a plurality of vertices of an irregular polygon on the calibration object.
(Item 12)
12. The method according to item 11, wherein the one or more reference markers include three reference markers that form a plurality of vertices of an isosceles or unequal triangle.
(Item 13)
The one or more reference markers form a plurality of vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes the step of determining the spatial polygon in the plurality of images. 12. The method of item 11, wherein a correspondence between a plurality of vertices and the plurality of vertices of the irregular polygon on the calibration object is identified.
(Item 14)
The method of item 1, wherein the plurality of features are arranged in a repeating pattern on a plane, and the one or more reference markers form a shape that is non-rotationally symmetric in the plane.
(Item 15)
The method of item 1, wherein the one or more imaging devices include a first camera and a second camera that capture a plurality of images substantially simultaneously.
(Item 16)
16. The method of item 15, wherein the formulating step comprises identifying a plurality of features present in both an image obtained by the first camera and a corresponding image obtained by the second camera.
(Item 17)
2. The method of item 1, wherein the plurality of calibration parameters include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices.
(Item 18)
The method of item 1, wherein the plurality of calibration parameters includes a position and orientation of each of the one or more imaging devices relative to the calibration object.
(Item 19)
A system for calibrating one or more imaging devices comprising:
One or more imaging devices each configured to capture a plurality of images to be calibrated including a plurality of features arranged in a repeating pattern and one or more reference markers;
With one or more processors,
Each of the one or more reference markers is uniquely identifiable in each of the plurality of images,
The plurality of images includes at least one image capturing only a part of all of the plurality of features,
The one or more processors may be collectively or individually,
Receiving the plurality of images from each of the one or more imaging devices;
Determining a spatial relationship between the one or more reference markers in each of the plurality of images and the one or more reference markers on the calibration object;
Based on the plurality of predetermined global coordinates of the one or more reference markers, image coordinates of a plurality of features in each image of the plurality of images, and global coordinates regarding a plurality of features corresponding to the calibration target, Formulate the correspondence between
A system that determines a plurality of calibration parameters for the one or more imaging devices based on the correspondence.
(Item 20)
20. A system according to item 19, wherein the one or more imaging devices are mounted by an unmanned aerial vehicle.
(Item 21)
20. The system of item 19, wherein each image of the plurality of images captures the one or more reference markers and at least a subset of the plurality of features on the calibration object.
(Item 22)
20. The system of item 19, wherein the plurality of images from each of the one or more imaging devices are imaged at different positions and orientations relative to the calibration object.
(Item 23)
Item 20. The system according to Item 19, wherein the calibration object is a checkerboard including a plurality of squares, and the plurality of features include a plurality of corner points of the plurality of squares.
(Item 24)
24. The system of item 23, wherein the one or more fiducial markers are each positioned within one of the plurality of squares.
(Item 25)
24. The system of item 23, wherein the plurality of features includes four outer corners of the checkerboard, and the at least one image excludes at least one of the four outer corners.
(Item 26)
20. The system of item 19, wherein the one or more reference markers includes at least three reference markers.
(Item 27)
20. The system of item 19, wherein the one or more fiducial markers comprise a plurality of circular dots, triangles, or quadrilaterals.
(Item 28)
20. The system of item 19, wherein the one or more reference markers are overlaid on the repeating pattern.
(Item 29)
20. The system of item 19, wherein the one or more reference markers form a plurality of irregular polygon vertices on the calibration object.
(Item 30)
30. The system of item 29, wherein the one or more fiducial markers include three fiducial markers that form a plurality of vertices of an isosceles or unequal triangle.
(Item 31)
The one or more reference markers form a plurality of vertices of an irregular polygon in the plurality of images, and determining the spatial relationship is a function of the irregular polygon in the plurality of images. 30. The system of item 29, wherein a correspondence between a plurality of vertices and the plurality of vertices of the irregular polygon on the calibration object is identified.
(Item 32)
20. The system of item 19, wherein the plurality of features are arranged in a repeating pattern on a plane, and the one or more reference markers form a shape that is non-rotationally symmetric in the plane.
(Item 33)
20. The system of item 19, wherein the one or more imaging devices include a first camera and a second camera that capture multiple images at substantially the same time.
(Item 34)
The formulation comprises identifying a plurality of features present in both the plurality of images obtained by the first camera and the corresponding plurality of images obtained by the second camera; 34. The system according to item 33.
(Item 35)
20. The system of item 19, wherein the plurality of calibration parameters include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices.
(Item 36)
20. The system of item 19, wherein the plurality of calibration parameters includes a respective position and orientation of the one or more imaging devices relative to the calibration object.
(Item 37)
A method for generating a plurality of depth maps using one or more imaging devices, comprising:
Obtaining one or more images from each of the one or more imaging devices;
With the assistance of one or more processors, the one or more images from each of the one or more imaging devices based on a plurality of calibration parameters for the one or more imaging devices to generate a three-dimensional depth map A processing stage, and
The plurality of calibration parameters are determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices,
The calibration object includes a plurality of features arranged in a repeating pattern and one or more reference markers,
Each of the one or more reference markers is uniquely identifiable within each calibration image of the plurality of calibration images;
The method wherein the plurality of calibration images includes at least one calibration image that captures only a portion of all of the plurality of features.
(Item 38)
38. The method of item 37, wherein the one or more imaging devices are carried by an unmanned aerial vehicle.
(Item 39)
40. The method of item 38, wherein the depth map includes information indicating a plurality of distances of one or more objects from the unmanned aerial vehicle.
(Item 40)
38. The method of item 37, wherein each calibration image of the plurality of calibration images captures the one or more reference markers and at least a subset of the plurality of features on the calibration object.
(Item 41)
40. The method of item 37, wherein the plurality of calibration images from each of the plurality of imaging devices are imaged at a plurality of different positions and orientations relative to the calibration object.
(Item 42)
38. A method according to item 37, wherein the calibration object is a checkerboard including a plurality of squares, and the plurality of features include a plurality of corner points of the plurality of squares.
(Item 43)
43. The method of item 42, wherein each of the one or more reference markers is positioned within one of the plurality of squares.
(Item 44)
43. The method of item 42, wherein the plurality of features includes four outer corners of the checkerboard, and the at least one calibration image excludes at least one of the four outer corners.
(Item 45)
38. The method of item 37, wherein the one or more reference markers includes at least three reference markers.
(Item 46)
38. The method of item 37, wherein the one or more reference markers comprise a plurality of circular dots, triangles, or quadrilaterals.
(Item 47)
38. The method of item 37, wherein the one or more reference markers are overlaid on the repeating pattern.
(Item 48)
38. The method of item 37, wherein the one or more reference markers form a plurality of irregular polygonal vertices on the calibration object.
(Item 49)
49. The method of item 48, wherein the one or more fiducial markers include three fiducial markers that form a plurality of vertices of an isosceles or unequal triangle.
(Item 50)
The one or more reference markers form a plurality of vertices of an irregular polygon in the plurality of calibration images, and the plurality of calibration parameters are at least the irregular polygons in the plurality of calibration images. 49. The method of item 48, wherein the method is determined by identifying a correspondence between the plurality of vertices and the plurality of vertices of the irregular polygon on the calibration object.
(Item 51)
38. The method of item 37, wherein the plurality of features are arranged in the repeating pattern on a plane, and the one or more reference markers form a shape that is non-rotationally symmetric in the plane.
(Item 52)
38. The method of item 37, wherein the one or more imaging devices include a first camera and a second camera that capture a plurality of images substantially simultaneously.
(Item 53)
Processing the one or more images comprises identifying one or more features present in both an image captured by the first camera and a corresponding image captured by the second camera; 53. The method according to item 52.
(Item 54)
53. The item 52, wherein processing the one or more images comprises determining parallax between an image captured by the first camera and a corresponding image captured by the second camera. the method of.
(Item 55)
38. The method of item 37, wherein the plurality of calibration parameters include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices.
(Item 56)
38. The method of item 37, wherein the plurality of calibration parameters include a position and orientation of each of the one or more imaging devices relative to the calibration object.
(Item 57)
38. The method of item 37, wherein processing the one or more images comprises reducing optical distortion in at least one of the one or more images.
(Item 58)
A system for generating a plurality of depth maps using one or more imaging devices comprising:
One or more imaging devices each configured to obtain a plurality of images;
With one or more processors,
The one or more processors may be collectively or individually,
Obtaining one or more images from each of the one or more imaging devices;
Configured to process the one or more images from each of the one or more imaging devices based on a plurality of calibration parameters for the one or more imaging devices to generate a three-dimensional depth map;
The plurality of calibration parameters are determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices,
The calibration object includes a plurality of features arranged in a repeating pattern and one or more reference markers,
Each of the one or more reference markers is uniquely identifiable within each calibration image of the plurality of calibration images;
The plurality of calibration images includes at least one calibration image that captures only a portion of all of the plurality of features.
(Item 59)
59. A system according to item 58, wherein the one or more imaging devices are carried by an unmanned aerial vehicle.
(Item 60)
60. The system of item 59, wherein the depth map includes information indicating a plurality of distances of one or more objects from the unmanned aerial vehicle.
(Item 61)
59. The system of item 58, wherein each calibration image of the plurality of calibration images captures the one or more reference markers and at least a subset of the plurality of features on the calibration object.
(Item 62)
59. The system of item 58, wherein the plurality of calibration images from each of the plurality of imaging devices are imaged at different positions and orientations relative to the calibration object.
(Item 63)
59. A system according to item 58, wherein the calibration object is a checkerboard including a plurality of squares, and the plurality of features include a plurality of corner points of the plurality of squares.
(Item 64)
64. The system of item 63, wherein the one or more fiducial markers are each positioned within one of the plurality of squares.
(Item 65)
64. The system of item 63, wherein the plurality of features includes four outer corners of the checkerboard, and the at least one calibration image excludes at least one of the four outer corners.
(Item 66)
59. The system of item 58, wherein the one or more reference markers include at least three reference markers.
(Item 67)
59. The system of item 58, wherein the one or more reference markers include a plurality of circular dots, triangles, or quadrilaterals.
(Item 68)
59. The system of item 58, wherein the one or more reference markers are overlaid on the repeating pattern.
(Item 69)
59. A system according to item 58, wherein the one or more reference markers form a plurality of vertices of an irregular polygon on the calibration object.
(Item 70)
70. A system according to item 69, wherein the one or more reference markers include three reference markers forming a plurality of vertices of an isosceles or unequal triangle.
(Item 71)
The one or more reference markers form a plurality of vertices of an irregular polygon in the plurality of calibration images, and the plurality of calibration parameters are at least the irregular polygon in the plurality of calibration images. 70. The system of item 69, determined by identifying correspondence between the plurality of vertices and the plurality of vertices of the irregular polygon on the calibration object.
(Item 72)
59. The system of item 58, wherein the plurality of features are arranged in the repeating pattern on a plane, and the one or more reference markers form a shape that is non-rotationally symmetric in the plane.
(Item 73)
59. The system of item 58, wherein the one or more imaging devices include a first camera and a second camera that capture a plurality of images substantially simultaneously.
(Item 74)
Processing the one or more images comprises identifying one or more features present in both an image captured by the first camera and a corresponding image captured by the second camera; 74. The system according to item 73.
(Item 75)
74. The step of processing the one or more images comprises determining a parallax between an image captured by the first camera and a corresponding image captured by the second camera. System.
(Item 76)
59. The system of item 58, wherein the plurality of calibration parameters includes at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices.
(Item 77)
59. The system of item 58, wherein the plurality of calibration parameters includes a respective position and orientation of the one or more imaging devices relative to the calibration object.
(Item 78)
59. The system of item 58, wherein processing the one or more images comprises reducing optical distortion in at least one of the one or more images.

Claims (37)

A method for generating a plurality of depth maps using one or more imaging devices, comprising:
Obtaining one or more images from each of the one or more imaging devices;
Processing the one or more images based on a plurality of calibration parameters for the one or more imaging devices to generate a three-dimensional depth map;
The plurality of calibration parameters are determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices;
The calibration object includes a plurality of features and a plurality of reference markers arranged in a repeating pattern,
Each of the plurality of fiducial markers is uniquely identifiable within each calibration image of the plurality of calibration images;
The plurality of reference markers form a plurality of vertices of a single polygon on the calibration object;
All of the edges of the single polygon have different lengths, or all of the angles of the plurality of vertices of the single polygon are different from each other;
The method wherein the plurality of calibration images includes at least one calibration image that captures only some of the plurality of features.   The method of claim 1, wherein the one or more imaging devices are mounted on an unmanned aerial vehicle. The method of claim 2, wherein the three-dimensional depth map includes information indicating a distance of one or more objects from the unmanned aerial vehicle.   4. The method according to claim 1, wherein each calibration image captures the plurality of reference markers and at least a subset of the plurality of features on the calibration object.   The method according to claim 1, wherein the plurality of calibration images are imaged at different positions and orientations relative to the calibration object.   The repeating pattern is a lattice pattern having a plurality of rows and a plurality of columns,
  The method according to claim 1, wherein the plurality of reference markers are positioned in different rows and columns in the grid pattern. The calibration object is a checkerboard including a plurality of square, the plurality of features includes a plurality of corner points of the plurality of square method according to any one of claims 1 to 6. The method of claim 7 , wherein each of the plurality of fiducial markers is positioned within one of the plurality of squares. 9. The method of claim 7 or 8 , wherein the plurality of features includes four outer corners of the checkerboard and the at least one calibration image excludes at least one of the four outer corners. Wherein the plurality of reference markers comprises at least three reference markers, the method according to any one of claims 1 to 9.   The plurality of features are arranged in the repetitive pattern on a plane, and the plurality of reference markers form a shape that is non-rotationally symmetric in the plane. Method. (I) The plurality of reference markers are overlaid on the repetitive pattern.
(Ii) the plurality of fiducial markers, single forming a plurality of vertices of scalene triangle, or (iii) said plurality of calibration parameters on the calibration object, at least, the in the plurality of calibration images single and the polygon of the plurality of vertices one, is determined by identifying the correspondence between said single plurality of vertices of the polygon on the calibration target, any one of claims 1 to 11 one The method according to item. 13. The method according to any one of claims 1 to 12 , wherein the one or more imaging devices include a first camera and a second camera that acquire a plurality of images simultaneously. Processing the one or more images comprises a step of identifying one or more features in the image corresponding acquired by the first acquired image and the second camera by the camera, according to claim 13 Method. Processing the one or more images comprises the step of determining the image acquired by the first camera, the parallax between the corresponding image obtained by the second camera, to claim 13 or 14 The method described. 16. The plurality of calibration parameters according to any one of claims 1 to 15 , wherein the plurality of calibration parameters include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices. The method described. Wherein the plurality of calibration parameters, including the positions and orientations of the one or more imaging devices relative to the calibration object, the method according to any one of claims 1 to 16. The method of any one of claims 1 to 17 , wherein processing the one or more images comprises reducing optical distortion in at least one of the one or more images. An apparatus for generating a plurality of depth maps using one or more imaging devices,
One or more imaging devices each configured to obtain a plurality of images;
With one or more processors,
The one or more processors are:
Acquiring one or more images from each of the one or more imaging devices;
Processing the one or more images based on a plurality of calibration parameters for the one or more imaging devices to generate a three-dimensional depth map;
The plurality of calibration parameters are determined using a plurality of calibration images to be calibrated from each of the one or more imaging devices;
The calibration object includes a plurality of features and a plurality of reference markers arranged in a repeating pattern,
Each of the plurality of fiducial markers is uniquely identifiable within each calibration image of the plurality of calibration images;
The plurality of reference markers form a plurality of vertices of a single polygon on the calibration object;
All of the edges of the single polygon have different lengths, or all of the angles of the plurality of vertices of the single polygon are different from each other;
The apparatus, wherein the plurality of calibration images includes at least one calibration image that captures only some of the plurality of features. The apparatus of claim 19 , wherein the one or more imaging devices are mounted on an unmanned aerial vehicle. 21. The apparatus of claim 20, wherein the three-dimensional depth map includes information indicating a distance of one or more objects from the unmanned aerial vehicle. 22. The apparatus according to any one of claims 19 to 21 , wherein each calibration image captures the plurality of reference markers and at least a subset of the plurality of features on the calibration object. 23. The apparatus according to any one of claims 19 to 22 , wherein the plurality of calibration images are imaged at different positions and orientations relative to the calibration object.   The repeating pattern is a lattice pattern having a plurality of rows and a plurality of columns,
  24. The apparatus according to any one of claims 19 to 23, wherein the plurality of reference markers are positioned in different rows and columns in the grid pattern. The apparatus according to any one of claims 19 to 24 , wherein the calibration object is a checkerboard including a plurality of squares, and the plurality of features include a plurality of corner points of the plurality of squares.   26. The apparatus of claim 25, wherein each of the plurality of reference markers is positioned within one of the plurality of squares. 27. The apparatus of claim 25 or 26 , wherein the plurality of features include four outer corners of the checkerboard and the at least one calibration image excludes at least one of the four outer corners. 28. The apparatus according to any one of claims 19 to 27 , wherein the plurality of fiducial markers includes at least three fiducial markers.   The plurality of features are arranged in the repeating pattern on a plane, and the plurality of reference markers form a shape that is non-rotationally symmetric in the plane. apparatus. (I) the one or more reference markers are overlaid on the repeating pattern;
(Ii) the plurality of fiducial markers form a plurality of vertices of an unequal triangle on the calibration object; or (iii) the plurality of calibration parameters at least in the single calibration image in the plurality of calibration images. said plurality of vertices of polygon is determined by identifying the correspondence between the single polygon of the plurality of vertices on the calibration target, according to any one of claims 19 to 29 Equipment. 31. The apparatus according to any one of claims 19 to 30 , wherein the one or more imaging devices include a first camera and a second camera that acquire a plurality of images simultaneously. Said processing of said one or more images includes identifying one or more features present in both of the corresponding image obtained by the first acquired image and the second camera by the camera, according to claim 31 The device described in 1. 33. The process of claim 31 or 32 , wherein the processing of the one or more images includes determining a parallax between an image acquired by the first camera and a corresponding image acquired by the second camera. Equipment. 34. The method of any one of claims 19 to 33 , wherein the plurality of calibration parameters include at least one of focal length, principal point location, pixel size, or optical distortion for each of the one or more imaging devices. The device described. 35. The apparatus according to any one of claims 19 to 34 , wherein the plurality of calibration parameters includes a respective position and orientation of the one or more imaging devices relative to the calibration object. 36. The apparatus according to any one of claims 19 to 35 , wherein the processing of the one or more images includes reducing optical distortion in at least one of the one or more images. A program for causing a computer to execute the method according to any one of claims 1 to 18 .