DE202014011514U1 - Camera calibration - Google Patents

Abstract

A system for calibrating imaging devices, the system comprising:
first and second imaging devices carried by an unmanned aerial vehicle, each imaging device configured to capture an image data set comprising a plurality of images (350, 352) of a calibration target (300, 500), the calibration target (300, 500) a plurality of features (302, 304) arranged in a repeating pattern and having a plurality of reference marks (502a, 502b, 502c), the reference marks (502a, 502b, 502c) in each image of the plurality of images (350, 352 ) are each uniquely identifiable, wherein the reference markings form vertices of an irregular polygon on the calibration target, wherein the one or more reference markings form vertices of an irregular polygon in the plurality of images and wherein the plurality of images (350, 352) comprises at least one image, that receives less than all of the plurality of features (302, 304); and
one or more processors that are configured jointly or individually to:
Receiving the plurality of images (350, 352) from each of the first and second imaging devices;
Determining a spatial relationship between the reference marks (502a, 502b, 502c) in each image of the plurality of images (350, 352) and the reference marks (502a, 502b, 502c) on the calibration target (300, 500),
wherein the step of determining the spatial relationship comprises a first correspondence between the vertices of the irregular polygon in the plurality identify images and the vertices of the irregular polygon on the calibration target, wherein identification of corresponding reference marks is performed based on the positioning of the reference marks relative to edges or angles of the polygon;
Formulating, based on the spatial relationship, a second correspondence between image coordinates of features in each image of the plurality of images (350, 352) and global coordinates for corresponding features on the calibration target (300, 500), based on the predetermined global coordinates of the reference marks (502a, 502b,
502c); and
Determine calibration parameters for the first and
second imaging device based on the second match.

Description

PRIOR ART OF THE INVENTION

Unmanned vehicles, such as unmanned aerial vehicles (UAVs), can be used to carry out surveillance, reconnaissance and reconnaissance tasks for a wide variety of military and civil applications. In some cases, a UAV can carry a camera to generate image data of objects in an environment. Based on the specific application, a camera calibration procedure can be performed to determine one or more parameters of the camera.

Existing approaches to camera calibration may not be optimal in some cases. For example, some camera calibration methods for calibrating imaging systems with multiple cameras may not be optimal.

SUMMARY OF THE INVENTION

The present disclosure provides systems, methods, and apparatus for improved calibration of cameras and other types of imaging devices. In some embodiments, the calibration techniques described herein utilize images obtained by one or more imaging devices of a calibration target that has a plurality of features and one or more uniquely identifiable fiducial marks. The reference marks can be used to facilitate determination of a correspondence between image coordinates of features in the images and the global coordinates of corresponding features on the calibration target. Based on the determined correspondence, one or more calibration parameters (e.g. extrinsic and / or intrinsic parameters) can be calculated for the imaging devices. Some aspects of the present disclosure allow calibration parameters to be determined using images that depict less than all of the calibration target (e.g., less than all of the features on the calibration target). This approach advantageously provides improved accuracy and flexibility of the calibration process, in particular with regard to a calibration of a plurality of imaging devices.

Therefore, in one aspect, a method for calibrating one or more imaging devices is provided. The method includes: receiving, from each of the one or more imaging devices, a plurality of images of a calibration target, the calibration target having a plurality of features established in a repeating pattern and one or more reference marks, the one or more a plurality of reference marks are each uniquely identifiable in each image of the plurality of images, and wherein the plurality of images includes at least one image that captures less than all of the plurality of features; Determining, using one or more processors, a spatial relationship between the one or more reference marks in each image of the plurality of images and the one or more reference marks on the calibration target; Formulating, based on the spatial relationship and with the aid of one or more processors, a correspondence between image coordinates of features in each image of the plurality of images and global coordinates for corresponding features on the calibration target; and determining, based on the match and with the aid of the one or more processors, calibration parameters for the one or more imaging devices.

In some embodiments, the one or more imaging devices are carried on an unmanned aerial vehicle.

In some embodiments, each image of the plurality of images captures the one or more reference marks and at least a subset of the plurality of features on the calibration target. The plurality of images from each of the one or more imaging devices can be captured at different positions and orientations relative to the calibration target.

In some embodiments, the calibration target is a checkerboard that includes a plurality of squares, and the plurality of features includes vertices of the plurality of squares. The one or more reference markings can each be positioned within one of the plurality of squares. The plurality of features can include four outside corners of the chessboard and the at least one image can omit at least one of the four outside corners.

In some embodiments, the one or more reference marks have at least three reference marks. The one or more reference markings can have circular points, triangles or squares. The one or more reference marks can be placed over the repeating pattern. The one or more reference marks may form vertices of an irregular polygon on the calibration target. The one or more reference markings can have three reference markings which form vertices of an isosceles or non-equilateral triangle.

In some embodiments, the one or more reference marks form vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes identifying a correspondence between the vertices of the irregular polygon in the plurality of images and the vertices of the irregular polygon on the Calibration target. The plurality of features can be established in the repeating pattern over a plane, and the one or more reference marks can form a shape that is rotationally asymmetrical within the plane.

In some embodiments, the one or more imaging devices include a first camera and a second camera that capture images substantially simultaneously. The formulating step may include identifying features that are present in both an image obtained from the first camera and a corresponding image obtained from the second camera. The calibration parameters may include at least one of a focal length, a position of the image center point, pixel size, or optical distortion for each of the one or more imaging devices. The calibration parameters may include a position and orientation of each of the one or more imaging devices relative to the calibration target.

In another aspect, a system for calibrating one or more imaging devices is provided. The system comprises: one or more imaging devices each configured to capture a plurality of images of a calibration target, the calibration target having a plurality of features arranged in a repeating pattern and one or more reference marks, the one or a plurality of reference marks are each uniquely identifiable in each image of the plurality of images, and wherein the plurality of images comprises at least one image that takes up less than all of the plurality of features; and one or more processors collectively or individually configured to: receive the plurality of images from each of the one or more imaging devices; Determining a spatial relationship between the one or more reference marks in each image of the plurality of images and the one or more reference marks on the calibration target; Formulating a correspondence between image coordinates of features in each image of the plurality of images and global coordinates for corresponding features on the calibration target based on the predetermined global coordinates of the one or more reference marks; and determining calibration parameters for the one or more imaging devices based on the match.

In some embodiments, the one or more imaging devices are carried on an unmanned aerial vehicle.

In some embodiments, each image of the plurality of images captures the one or more reference marks and at least a subset of the plurality of features on the calibration target. The plurality of images from each of the one or more imaging devices can be captured at different positions and orientations relative to the calibration target.

In some embodiments, the calibration target is a checkerboard that has a plurality of squares, and the plurality of features includes vertices of the plurality of squares. The one or more reference markings can each be positioned within one of the plurality of squares. The plurality of features can include four outside corners of the chessboard and the at least one image can omit at least one of the four outside corners.

In some embodiments, the one or more reference marks have at least three reference marks. The one or more reference markings can have circular points, triangles or squares. The one or more reference marks can be placed over the repeating pattern. The one or more reference marks may form vertices of an irregular polygon on the calibration target. The one or more Reference marks can have three reference marks that form vertices of an isosceles or unequal-sided triangle.

In some embodiments, the one or more reference marks form vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes identifying a correspondence between the vertices of the irregular polygon in the plurality of images and the vertices of the irregular polygon on the Calibration target. The plurality of features can be established in the repeating pattern over a plane, and the one or more reference marks can form a shape that is rotationally asymmetrical within the plane.

In some embodiments, the one or more imaging devices include a first camera and a second camera that capture images substantially simultaneously. The formulating step may include identifying features that are present in both an image obtained from the first camera and a corresponding image obtained from the second camera. The calibration parameters may include at least one of a focal length, a position of the image center point, pixel size, or optical distortion for each of the one or more imaging devices. The calibration parameters may include a position and orientation of each of the one or more imaging devices relative to the calibration target.

In another aspect, a method for generating depth maps using one or more imaging devices is provided. The method includes: obtaining one or more images from each of the one or more imaging devices; and processing, using one or more processors, the one or more images from each of the one or more imaging devices based on calibration parameters for the one or more imaging devices to generate a three-dimensional depth map, the calibration parameters using a plurality of calibration images of a calibration target can be determined by each of the one or more imaging devices, the calibration target having a plurality of features arranged in a repeating pattern and one or more reference marks, the one or more reference marks each being uniquely within each calibration image of the plurality of Calibration images are identifiable, and wherein the plurality of calibration images includes at least one calibration image that captures less than all of the plurality of features.

In some embodiments, the one or more imaging devices are carried on an unmanned aerial vehicle. The depth map can have information indicating the distances of one or more objects from the unmanned aerial vehicle.

In some embodiments, each calibration image of the plurality of calibrations includes the one or more reference marks and at least a subset of the plurality of features on the calibration target. The plurality of calibration images from each of the imaging devices can be captured at different positions and orientations relative to the calibration target.

In some embodiments, the calibration target is a checkerboard that has a plurality of squares, and the plurality of features includes vertices of the plurality of squares. The one or more reference markings can each be positioned within one of the plurality of squares. The plurality of features can include four outside corners of the chessboard and the at least one calibration image can omit at least one of the four outside corners.

In some embodiments, the one or more reference marks have at least three reference marks. The one or more reference markings can have circular points, triangles or squares. The one or more reference marks can be placed over the repeating pattern. The one or more reference marks may form vertices of an irregular polygon on the calibration target. The one or more reference markings can have three reference markings which form vertices of an isosceles or non-equilateral triangle.

In some embodiments, the one or more reference marks form vertices of an irregular polygon in the plurality of images and the calibration parameters are determined by at least one identifying a match between the vertices of the irregular polygon in the plurality of calibration images and the vertices of the irregular polygon determined by the calibration target. The plurality of features can be established in the repeating pattern over a plane, and the one or more fiducial marks can form a shape that is rotationally asymmetrical within the plane.

In some embodiments, the one or more reference marks form vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes identifying a correspondence between the vertices of the irregular polygon in the plurality of images and the vertices of the irregular polygon on the Calibration target. The plurality of features can be established in the repeating pattern over a plane, and the one or more reference marks can form a shape that is rotationally asymmetrical within the plane.

In some embodiments, the one or more imaging devices include a first camera and a second camera that capture images substantially simultaneously. Processing the one or more images may include identifying features present in both an image obtained from the first camera and a corresponding image obtained from the second camera. Processing the one or more images may include determining a difference between an image captured by the first camera and a corresponding image captured by the second camera. The calibration parameters may include at least one of a focal length, a position of the image center point, pixel size, or optical distortion for each of the one or more imaging devices. The calibration parameters may include a position and orientation of each of the one or more imaging devices relative to the calibration target. Processing the one or more images may include reducing optical distortion in at least one of the one or more images.

In another aspect, a system for generating depth maps using one or more imaging devices is provided. The system may include: one or more imaging devices each configured to obtain images; and one or more processors collectively or individually configured to: obtain, from each of the one or more imaging devices, one or more images; and processing the one or more images from each of the one or more imaging devices based on calibration parameters for the one or more imaging devices to generate a three-dimensional depth map, the calibration parameters using a plurality of calibration images of a calibration target from each of the one or more Imaging devices are determined, wherein the calibration target has a plurality of features that are set up in a repetitive pattern, and one or more reference marks, wherein the one or more reference marks are each uniquely identifiable within each calibration image of the plurality of calibration images and wherein the plurality of Calibration images comprises at least one calibration image that captures less than all of the plurality of features.

In some embodiments, the one or more imaging devices are carried on an unmanned aerial vehicle. The depth map can have information indicating the distances of one or more objects from the unmanned aerial vehicle.

In some embodiments, each calibration image of the plurality of calibrations includes the one or more reference marks and at least a subset of the plurality of features on the calibration target. The plurality of calibration images from each of the imaging devices can be captured at different positions and orientations relative to the calibration target.

In some embodiments, the calibration target is a checkerboard that has a plurality of squares, and the plurality of features includes vertices of the plurality of squares. The one or more reference marks can each be positioned within one of the plurality of squares. The plurality of features can include four outside corners of the chessboard and the at least one calibration image can omit at least one of the four outside corners.

In some embodiments, the one or more reference marks have at least three reference marks. The one or more reference markings can have circular points, triangles or squares. The one or more reference marks can be placed over the repeating pattern. The one or more reference marks may form vertices of an irregular polygon on the calibration target. The one or more Reference marks can have three reference marks that form vertices of an isosceles or unequal-sided triangle.

In some embodiments, the one or more reference marks form vertices of an irregular polygon in the plurality of images and the calibration parameters are determined by at least one identifying a match between the vertices of the irregular polygon in the plurality of calibration images and the vertices of the irregular polygon on the calibration target. The plurality of features can be established in the repeating pattern over a plane, and the one or more fiducial marks can form a shape that is rotationally asymmetrical within the plane.

In some embodiments, the one or more reference marks form vertices of an irregular polygon in the plurality of images, and the step of determining the spatial relationship includes identifying a correspondence between the vertices of the irregular polygon in the plurality of images and the vertices of the irregular polygon on the Calibration target. The plurality of features can be established in the repeating pattern over a plane, and the one or more reference marks can form a shape that is rotationally asymmetrical within the plane.

In some embodiments, the one or more imaging devices include a first camera and a second camera that capture images substantially simultaneously. Processing the one or more images may include identifying features present in both an image obtained from the first camera and a corresponding image obtained from the second camera. Processing the one or more images may include determining a difference between an image captured by the first camera and a corresponding image captured by the second camera. The calibration parameters may include at least one of a focal length, a position of the image center point, pixel size, or optical distortion for each of the one or more imaging devices. The calibration parameters may include a position and orientation of each of the one or more imaging devices relative to the calibration target. Processing the one or more images may include reducing optical distortion in at least one of the one or more images.

It should be understood that different aspects of the invention can be implemented individually, collectively, or in combination with one another. Various aspects of the invention described herein can be applied to any of the specific applications set forth below or to any other types of moving objects. Any description herein of an aircraft can be applied to and used for any moving object, such as any vehicle. In addition, the systems, devices and methods disclosed herein can also be applied in the context of air movement (e.g. flight) in the context of other types of movement, such as movement on the ground or on water, underwater movement or movement in space. Moreover, any description of a rotor or rotor assembly herein can be applied to any drive system, drive device, or drive mechanism and used to generate a drive force by rotation (e.g., propellers, wheels, axles).

Other objects and features of the present invention will become apparent upon a review of the specification, claims and appended figures.

RECORDING BY REFERENCE

All publications, patents, and patent applications mentioned in this application are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated herein by reference.

Figure list

• 1 ein Modell einer monokularen Kamera in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 2 ein Modell einer Zuordnung von einem Weltkoordinatensystem zu einem Bildkoordinatensystem in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 3 ein Modell einer Zuordnung von einem Weltkoordinatensystem zu einem Bildkoordinatensystem in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 4 ein Kalibrierungsziel, das zur Verwendung bei Abbildungsvorrichtungskalibrierung geeignet ist, in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 5 Bilder eines Schachbrettkalibrierungsziels in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 6 und 7 Referenzmarkierungen, die basierend auf relativer Positionierung eindeutig identifizierbar sind, in Übereinstimmung mit Ausführungsformen veranschaulichen;
• 8 ein Kalibrierungsziel, das Referenzmarkierungen aufweist, in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 9 ein Verfahren zum Kalibrieren einer oder mehrerer Abbildungsvorrichtungen unter Verwendung eines Kalibrierungsziels mit einer oder mehreren Referenzmarkierungen in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 10 bis 13 eine Identifikation von Merkmalen und Referenzmarkierungen eines Kalibrierungsziels in einem Bild in Übereinstimmung mit Ausführungsformen veranschaulichen;
• 14 ein Modell räumlicher Beziehungen zwischen Koordinatensystemen bei binokularer Kamerakalibrierung in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 15 ein Modell einer Zuordnung zwischen verzerrten Bildkoordinaten und unverzerrten Bildkoordinaten in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 16 und 17 eine Bildrektifizierungsprozedur für ein binokulares Kamerasystem in Übereinstimmung mit Ausführungsformen veranschaulichen;
• 18 binokulare Kamerabilder nach einem Verarbeiten zum Durchführen von Bildrektifizierung in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 19 ein Verfahren zum Erzeugen von Tiefenkarten aus Bilddaten in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 20 ein UAV in Übereinstimmung mit Ausführungsformen veranschaulicht;
• 21 ein bewegliches Objekt, das einen Träger und eine Traglast aufweist, in Übereinstimmung mit Ausführungsformen veranschaulicht; und
• 22 ein System zum Steuern eines beweglichen Objekts in Übereinstimmung mit Ausführungsformen veranschaulicht.

The essential features of the novel features of the invention are set out in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description containing illustrative embodiments, in which the principles of the inventions are used, sets out, and receives the accompanying drawings, of which:

• 1 illustrates a model of a monocular camera in accordance with embodiments;
• 2 illustrates a model of mapping from a world coordinate system to an image coordinate system in accordance with embodiments;
• 3 illustrates a model of mapping from a world coordinate system to an image coordinate system in accordance with embodiments;
• 4th illustrates a calibration target suitable for use in imaging device calibration, in accordance with embodiments;
• 5 Illustrates images of a checkerboard calibration target in accordance with embodiments;
• 6th and 7th Illustrate fiducial marks that are uniquely identifiable based on relative positioning in accordance with embodiments;
• 8th illustrates a calibration target having fiducial marks in accordance with embodiments;
• 9 Illustrates a method of calibrating one or more imaging devices using a calibration target with one or more fiducial marks, in accordance with embodiments;
• 10 until 13th illustrate identification of features and fiducial marks of a calibration target in an image in accordance with embodiments;
• 14th illustrates a model of spatial relationships between coordinate systems with binocular camera calibration in accordance with embodiments;
• 15th illustrates a model of an association between distorted image coordinates and undistorted image coordinates in accordance with embodiments;
• 16 and 17th illustrate an image rectification procedure for a binocular camera system in accordance with embodiments;
• 18th illustrates binocular camera images after processing to perform image rectification in accordance with embodiments;
• 19th Illustrates a method for generating depth maps from image data in accordance with embodiments;
• 20th Illustrates a UAV in accordance with embodiments;
• 21 FIG. 11 illustrates a moving object including a carrier and a load in accordance with embodiments; FIG. and
• 22nd illustrates a system for controlling a moving object in accordance with embodiments.

DETAILED DESCRIPTION

The present disclosure provides improved systems, methods, and apparatus for performing calibration of imaging devices such as cameras. The approaches described herein may involve using one or more imaging devices to obtain a plurality of images of a calibration target. The calibration target can have a plurality of features and one or more reference marks. The one or more reference marks can be uniquely identifiable to facilitate a determination of the correspondence between image coordinates of the features in the images and world coordinates of the features on the calibration target, even if some of the images do not capture all of the features of the calibration target. This approach can be advantageous in providing accurate and reliable calibration even in situations where it is difficult for each of the imaging devices to obtain image data from the entirety of the calibration target, e.g., when multiple imaging devices are being calibrated at the same time. In addition, such approaches allow the use of larger calibration targets, which can improve the accuracy of the resulting calibration parameters.

The techniques described herein can be implemented using a wide variety of applications involving imaging and imaging data, such as vision-based navigation using one or more imaging devices carried by a moving object. As an example, the systems, methods, and apparatus of the present disclosure can be used to calibrate one or more cameras carried by an unmanned aerial vehicle (UAV). Each of the one or more cameras can be used to capture a plurality of images of a checkerboard calibration target having at least three circular fiducial marks superimposed on the squares of the checkerboard. The reference markings can be clearly identifiable in the images in order to facilitate the determination of the world coordinates of the corner points of the chessboard corresponding to the corner points that are recorded in the image data. This compliance information can be used to determine calibration parameters for the UAV cameras. Subsequent image data recorded by the cameras can be processed based on the calibration parameters, for example in order to generate three-dimensional (3D) depth maps of the surrounding environment and / or of objects within the environment.

The techniques described herein can be used to calibrate one or more imaging devices (e.g., cameras, scanners, etc.). An imaging device, which may also be referred to herein as a "vision sensor", may be configured to detect electromagnetic radiation (e.g., visible, infrared, and / or ultraviolet light) and generate image data based on the detected electromagnetic radiation. For example, an imaging device may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor that generates electrical signals in response to wavelengths of light. The resulting electrical signals can be processed to create image data. The image data generated by an imaging device can include one or more images, which can be static images (e.g. photographs), dynamic images (e.g. video), or any suitable combination thereof. The image data can be polychrome (e.g. RGB, CMYK, HSV) or monochrome (e.g. gray scale, black and white, sepia).

The imaging devices of the present disclosure can be carried by various types of objects, such as one or more of the movable objects described herein. An imaging device can be on any suitable portion of the object, such as above, below, on the side (s) of or within the object. Some imaging devices can be mechanically coupled to the object, so that the spatial arrangement and / or movement of the object correspond to the spatial arrangement and / or movement of the imaging devices. The imaging device can be coupled to the object via a rigid coupling so that the imaging device does not move relative to the portion of the object to which it is attached. Alternatively, the coupling between the imaging device and the object may allow movement of the imaging device relative to the object. The coupling can be a permanent coupling or a non-permanent (e.g. releasable) coupling. Suitable methods of coupling may include glue, bonding, welding, and / or fasteners (e.g. screws, nails, pins, etc.). Optionally, the imaging device can be formed integrally with a section of the object. In addition, the imaging device can be electrically coupled to a section of the object (e.g. processing unit, control system, data memory) in order to enable the data collected by the imaging device to be used for various functions of the object (e.g. navigation, control, drive, Communication with a user or other device, etc.), such as the embodiments discussed herein, may be used.

In some embodiments, the imaging devices are carried by a UAV. For example, the UAV can be a small format UAV (e.g. weighs no more than 10 kg, has a maximum dimension of no more than 1.5 m). The UAV may be a rotorcraft, such as a multi-rotor aircraft that is powered by a variety of propellers to move through the air (e.g., a quadrocopter). The UAVs described herein can be operated completely autonomously (for example by a suitable computing system such as a built-in controller), semiautonomously or manually (for example by a human user). The UAV can receive commands from a suitable entity (e.g. human user or autonomous control system) and respond to such commands by performing one or more actions. For example, the UAV can be controlled to take off from the ground, move in the air (e.g. with up to three degrees of freedom for translation and up to three degrees of freedom for rotation), to move to the target location or to a sequence of target locations in which Floating in the air, landing on the ground and so on. As another example, the UAV can be controlled to move at a particular speed and / or acceleration (e.g., with up to three degrees of freedom in translation and up to three degrees of freedom in rotation) or along a particular path of movement. In addition, the commands can be used to control one or more UAV components, such as the components described herein (e.g. sensors, actuators, propulsion units, payload, etc.).

In some embodiments, the imaging device can be a camera. While certain embodiments provided herein are described in the context of cameras, it should be understood that the present disclosure can be applied to any suitable imaging device, and any description herein relating to cameras can also be applied to other types of imaging devices. A camera can be used to generate 2D images of a 3D scene (e.g. an environment, one or more objects, etc.). The images generated by the camera can represent the projection of the 3D scene onto a 2D image plane. Accordingly, each point in the 2D image corresponds to a spatial 3D coordinate in the scene.

1 illustrates a model 100 a single camera (monocular camera) in accordance with embodiments. The model 100 has a world coordinate system 102 , a camera coordinate system 104 , an image plane coordinate system 106 and a pixel coordinate system 108 on. The world coordinate system 102 , which can also be referred to herein as a “global coordinate system”, represents the frame of reference of the scene recorded by the camera and / or the environment in which the camera operates. The world coordinate system 102 has a coordinate origin O _w and 3D coordinate axes X _w , Y _w , Z _w . In some embodiments, the world coordinate system 102 can be defined in relation to an object of interest in the scene, such as a calibration target, as described in more detail below. The camera coordinate system 104 represents the frame of reference of the camera and has a coordinate origin O _c and 3D coordinate axes X _c , Y _c , Z _c . The axis Z _c can correspond to the optical axis of the camera, the outward direction being the positive direction.

The image plane coordinate system 106 and pixel coordinate system 108 , which can also be referred to herein as “image coordinate systems”, represent the frame of reference of the image generated by the camera. The image plane coordinate system has a coordinate origin O _p and 2D coordinate axes X _p , Y _p . The locations of the pixels within the image can be in the sense of a pixel coordinate system 108 which has a coordinate origin at the upper left corner of the image and 2D coordinate axes u, v. The u-axis can be parallel to the image rows, with the right direction being the positive direction, while the v-axis can be parallel to the image columns, with the downward direction being the positive direction. In some embodiments, the is the camera coordinate system 104 with the image plane coordinate system 106 and pixel coordinate system 108 aligned. For example, the X _c axis of the camera coordinate system 104 parallel to the rows of images and in the same direction as the u-axis of the pixel coordinate system 108 be aligned. _{The Y c} axis of the camera coordinate system can be similar 104 parallel to the image columns and in the same direction as the v-axis of the pixel coordinate system 108 be aligned.

Even though 1 imaging a single imaging device, it should be understood that the embodiments described herein may be applied to any suitable number of imaging devices, such as one, two, three, four, five, or more imaging devices. The imaging devices can all be of the same type. Alternatively, at least some of the imaging devices can be of different types. The imaging devices can be positioned and oriented relative to one another as desired. For example, various embodiments provided herein utilize two imaging devices (binocular vision), e.g., a left camera and a right camera. The imaging devices can be in different positions and orientations such that the field of view of each imaging device is different. The fields of view of the imaging devices can overlap. In some embodiments, the imaging devices each provide image data of the same scene from different positions and / or orientations. The imaging devices may be configured to capture images of a scene at or about the same time. Alternatively, some imaging devices may be configured to capture images at different times than other imaging devices.

In some embodiments, it may be desirable to obtain 3D information of a scene using 2D image data of the scene collected from one or more imaging devices. For example, various applications may involve a 3D reconstruction (e.g. a 3D model such as a point cloud, an occupation grid, a depth map, disparity map, etc.) of a scene based on a plurality of 2D images of the scene (e.g. images captured at a variety of different positions and orientations). In embodiments where a plurality of imaging devices are used, the 3D reconstruction can be based on disparities between image data obtained from imaging devices located at different positions and / or orientations, as will be described in more detail below. Alternatively or in combination, 3D information can also be restored from image data obtained from a single imaging device, for example using structure-of-motion or other suitable techniques. The resulting 3D reconstruction can be used to determine different types of quantitative information regarding the scene, for example spatial positions of objects (e.g. distance from the camera, other objects), object dimensions (e.g. length, width, height). This information can be useful in a wide variety of applications, e.g. computer vision, object recognition, tracking, posture estimation, self-movement detection, etc. For example, in the context of UAV operations, scene reconstruction for functionalities such as navigation, obstacle avoidance, assignment, target tracking and the like can be helpful.

In some embodiments, the scene reconstruction techniques described herein involve determining the correspondence between the image coordinates in an image of a scene and the world coordinates (also known as global coordinates) of the actual scene. This correspondence can be identified by determining the conversions or assignments between two different coordinate systems, e.g. the world coordinate system, camera coordinate system and image coordinate system. The mathematical relationships that define these associations can be determined based on models of the scene, imaging devices and image plane, e.g., a pinhole camera model.

2 illustrates a model 200 an assignment of a world coordinate system to an image coordinate system in accordance with embodiments. The mapping model 200 maps the relationships between the world, camera and image coordinate system. The mapping model 200 can be used to describe the mapping process (e.g. projecting world coordinates of an object in a scene onto image coordinates in an image) and the reconstruction process (e.g. using image coordinates in an image to recreate the world coordinates of an object in a scene).

wobei R die Rotationsmatrix ist und T der Translationsvektor ist, der angibt, wie das Kamerakoordinatensystem relativ zu dem Weltkoordinatensystem ausgerichtet beziehungsweise positioniert ist. Die Kamerakoordinaten können skaliert werden, um die entsprechenden Bildebenenkoordinaten zu erhalten: $$\{\begin{array}{c}{X}_p=X_c/Z_c\\ Y_p=Y_c/Z_c\end{array}$$

The world coordinate system can refer to the camera coordinate system, through the equation $$\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\end{array}\right]=\mathrm{R.}\cdot \left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+T$$

where R is the rotation matrix and T is the translation vector, which indicates how the camera coordinate system is oriented or positioned relative to the world coordinate system. The camera coordinates can be scaled to get the corresponding image plane coordinates: $$\{\begin{array}{c}{X}_p=X_c/Z_c\\ Y_p=Y_c/Z_c\end{array}$$

wobei $r=\sqrt{X_p^2+Y_p^2}$

ist und k₁, k₂, k₃, p₁, p₂, s₁, s₂ die Verzerrungsparameter sind, k₁, k₂, k₃) die radialen Verzerrungsparameter sind, p₁, p₂ die exzentrischen Verzerrungsparameter sind, und wobei s₁, s₂ die Prismenverzerrungsparameter sind. In manchen Ausführungsformen werden nur vier der Parameter (z.B. k₁, k₂, p₁, p₂) berechnet.In embodiments where the camera has optical lens distortion (e.g., radial distortion, eccentric distortion, thin prism distortion), the image plane coordinates with distortion X _d , Y _d can be determined from the image plane coordinates using techniques known to those skilled in the art, such as from the following distortion equation : $$\{\begin{array}{c}{X}_d=X_p\left(1+k_1{r}^2+k_2{r}^{\mathrm{4th}}+k_3{r}^{\mathrm{6th}}+\cdots \right)+2_{p_1}{X}_p{Y}_p+p_2\left(r^2+2X_p^2\right)+s_1{r}^2\\ Y_d=Y_c\left(1+k_1{r}^2+k_2{r}^{\mathrm{4th}}+k_3{r}^{\mathrm{6th}}+\cdots \right)+p_1\left(r^2+2Y_p^2\right)+2_{p_2}{X}_p{Y}_p+s_2{r}^2\end{array}$$

whereby $r=\sqrt{X_p^2+Y_p^2}$

and k ₁ , k ₂ , k ₃ , p ₁ , p ₂ , s ₁ , s _{2 are} the distortion parameters, k ₁ , k ₂ , k ₃ ) are the radial distortion parameters, p ₁ , p _{2 are} the eccentric distortion parameters, and where s ₁ , s _{2 are} the prism distortion parameters. In some embodiments, only four of the parameters (eg, k ₁ , k ₂ , p ₁ , p ₂ ) are calculated.

wobei M die intrinsische Kameramatrix ist, u_o, v_o die Koordinaten des Bildmittelpunkts sind (wo die optische Achse die Bildebene schneidet, auch als der Bildmittelpunkt bekannt), und $\{\begin{array}{c}{f}_x=f/a_x\\ f_y=f/a_y\end{array}\text{,}$

ist, wobei f die Brennweite der Kamera ist und a_x, a_y die horizontalen beziehungsweise vertikalen Pixelabmessungen sind.The pixel coordinates can be obtained using the following relationship: $$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathrm{M.}\cdot \left[\begin{array}{c}{X}_d\\ Y_d\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& {\mathrm{<figure-callout\; id="0"\; label="u"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">u</figure-callout>}}_0\\ 0& {\mathrm{<figure-callout\; id="0"\; label="f\; y\; v"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">f</figure-callout>}}_y& v_{}{}_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_d\\ Y_d\\ 1\end{array}\right]$$

where M is the intrinsic camera matrix, u _o , v _{o are} the coordinates of the image center point (where the optical axis intersects the image plane, also known as the image center point), and $\{\begin{array}{c}{f}_x=f/a_x\\ f_y=f/a_y\end{array}\text{,}$

where f is the focal length of the camera and a _x , a _{y are} the horizontal and vertical pixel dimensions, respectively.

3 illustrates a model 250 an assignment of a world coordinate system to an image coordinate system in accordance with embodiments. The relationship between the world coordinate system, camera coordinate system, undistorted image plane coordinate system, distorted image plane coordinate system, and pixel coordinate system can be similar to those herein with respect to the mapping model 200 from 2 be described. Similar to assignment model 2, the assignment model 250 can be used to describe the mapping process (e.g. projecting world coordinates (X _w , Y _w , Z _w ) onto pixel coordinates (u, v)). The world coordinates of an object can be restored from the pixel coordinates by de-warping the image to get undistorted pixel coordinates (u ', v') and then reconstructing the world coordinates using the undistorted pixel coordinates.

In order to accurately perform scene restoration from image data, it may be advantageous to determine or estimate one or more parameters of the imaging devices that are associated with the associations described herein. These parameters can be extrinsic parameters (the rotation matrix R and the translation vector T, which indicate how each imaging device is positioned and oriented relative to the calibration target) and intrinsic parameters (the intrinsic matrix M, the information regarding the focal length, position of the image center, the pixel size having optical distortion parameters k ₁ , k ₂ , k ₃ , p ₁ , p ₂ , s ₁ , s ₂ ). In embodiments where a plurality of imaging devices are used, the extrinsic parameters may also include the rotation matrices and translation vectors for converting between the camera coordinate systems of the imaging devices that indicate how the imaging devices are positioned and oriented relative to one another, as will be described in more detail below. The process of obtaining the extrinsic and / or intrinsic parameters of one or more imaging devices may be referred to herein as "camera calibration".

Camera calibration can be particularly useful when the imaging devices are mounted on or carried by a moving object, such as a UAV. The operation of such moving objects can alter changes in the position and / or orientation of the imaging devices, for example due to vibrations and other movements that can affect the calibration parameters. Calibration of imaging devices carried by a moving object can occur while the moving object is operating (online calibration) and / or while the moving object is not operating (offline calibration). For example, offline calibration for cameras carried by a UAV can be performed when the UAV is powered off and / or not flying, while online calibration can be performed while the UAV is powered on and / or flying.

Camera calibration can be done in a number of ways. In some embodiments, the calibration process involves obtaining images of a scene that have a plurality of features, then identifying matches between image coordinates of the features in the images and the world coordinates of the corresponding features in the scene. The calibration parameters can then be estimated based on the identified matches and the mapping between the world and image coordinate systems (e.g., based on the mapping models described herein). For example, monocular calibration (determining the parameters of a single camera) may involve generating an estimate of the calibration parameters (e.g. R, T, M) based on a pinhole camera model, then estimating the distortion effect (e.g. initial values of distortion parameters are assumed to be 0), then perform a parameter optimization (e.g. using the Levenberg-Marquardt algorithm) in order to obtain the optimal camera calibration parameters.

In some embodiments, a calibration target is used to facilitate the identification of worldview coordinate matches. A calibration target can be any structure that has a defined arrangement of features. For example, the calibration target may be 1D (e.g., a linear, curved, or curvilinear shape), 2D (e.g., a plane or other flat surface), or 3D (e.g., a cube, sphere, prism, two or more intersecting planes). The features can be designed to be easily identifiable from image data (e.g. using image recognition algorithms, feature extraction algorithms, or other computer vision techniques) and clearly distinguishable from non-features (e.g. other sections of the calibration target, other objects in the scene, image noise). In some embodiments, the features are points on the calibration target (eg corners, vertices, intersections), also referred to herein as “feature points”. Alternatively or in combination, the features can be lines (e.g. linear, curved, curvilinear), edges (e.g. between two differently colored areas), geometric shapes (e.g. circles, triangles, squares, etc.), symbols or characters (e.g. alphanumeric characters) or suitable Be combinations of these. In addition, the characteristics of each feature (e.g. size, shape, color) can be varied as desired.

The calibration target can have any suitable number and combination of features. For example, a calibration target can have at least 10, 20, 30, 40, 50, 60, 70, 80, 90, or 100 features. Using a relatively large number of features can improve the accuracy of the calibration process. In some embodiments, a calibration target can have only one type of feature, while in other embodiments the target can have more than one type of feature, such as at least two, three, four, five or more different types of features. The features can lie at defined locations on the calibration target, so that the position and / or orientation of each feature relative to the calibration target are known. In some embodiments, the features on the calibration target are arranged in a repeating pattern (e.g., a grid having a plurality of rows and columns) that has predetermined dimensions (e.g. length, width, height, spacing between features). The features can be static features so that their position and / or orientation on the calibration target does not change. Alternatively, the features can be dynamic features with varying positions and / or orientations. The features described herein can be formed integrally with the calibration target (e.g., printed or painted on a portion of the target) or provided separately and coupled to the target (e.g., attached or attached to a portion of the target).

In some embodiments, the calibration process involves capturing a set of images of the calibration target with each imaging device to be calibrated. Each image data set can have at least 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 or more images of the calibration target. The images can depict the calibration target in a variety of different positions and / or orientations. Each image can be processed to identify features present in the image and determine their image coordinates. The correspondences between image coordinates of the features in the image data and the world coordinates of the features on the calibration target can be determined and subsequently used to solve the calibration parameters (e.g. based on the mathematical relationships described herein). Optionally, in embodiments where multiple imaging devices are used to obtain multiple sets of image data, the correspondences between image coordinates of features across the image data sets can also be determined and used to calculate the parameters for the multiple imaging devices (e.g. position and / or orientation of imaging devices relative to each other).

4th illustrates a calibration target 300 suitable for use in imaging device calibration, in accordance with embodiments. The goal 300 includes a flat checkerboard pattern that is a repeating pattern of black squares 302 alternating with white squares 304 having. Each of the squares 302 , 304 may be identically sized and arranged in a grid having a plurality of rows and columns. In some embodiments, the vertices serve as the squares (e.g., such as by circles 305 as the feature points of the calibration target 300 to perform imaging device calibration. Corner points can be relatively easily distinguishable from other sections of the image data using machine vision methods. Alternatively, or in combination, other sections of the chessboard can be used as the features of the target 300 serve for calibration purposes (e.g. the black squares 302 who have favourited white squares 304 etc.). The characteristics of the target 300 can from image data of the calibration target 300 identified using various feature extraction techniques and the image coordinates of each feature determined. This process can be accomplished by knowing the geometry of the calibration target 300 be relieved. For example, in embodiments where the corner points of the checkerboard squares 302 , 304 than the feature points of the calibration target 300 serve, the size of the squares defines the distance between adjacent feature points. The image coordinates each corner point can therefore be determined based on their row and column position within the chessboard.

Once the image coordinates of the features in the image data of the target 300 determined, the correspondence between the image coordinates and the world coordinates of the features on the target 300 be identified. The world coordinate system can be relative to the destination 300 can be defined so that the world coordinates of each feature indicate the position of the feature on the target 300 correspond. For example, the origin O _{w of} the world coordinate system can be at one of the four outside corners ( 306a , 306b , 306c , 306d ) of the chessboard, the plane of the chessboard _{lying along the X w} , Y _w plane of the world coordinate system. Accordingly, in embodiments where the vertices serve as the features for calibration, the Z _w world coordinate of each vertex on the target 300 is 0 and the X _w , Y _w world coordinate of each vertex can be based on its 2D location (e.g. row and column position) to be determined on the chessboard.

5 illustrates first and second images 350 , 352 of a checkerboard calibration target similar to the target 300 , in accordance with embodiments. In the embodiment of 5 the calibration target is an image displayed on a screen. However, it should be understood that the calibration targets described herein can be provided in other formats, such as printed on a material (e.g., a sheet of paper). The first picture 350 and second picture 352 of the calibration target may be obtained from a left camera and a right camera of binocular and stereo vision imaging systems, respectively, as part of a set of calibration image data for performing camera calibration. In particular, take both the first and second image 350 , 352 the entirety of the calibration target (e.g. the four outer corners of the chessboard, the four outer edges of the chessboard, all of the black and white squares) so that all the features of the calibration target (e.g. all of the corner points of the chessboard) are within both images 350 , 352 are visible. Correspondingly, the correspondences between image coordinates of the features (for example corner points) in the first image 350 , Image coordinates of corresponding features in the second image 352 and the world coordinates of the features in the calibration target are identified and used to determine calibration parameters for the left and right cameras.

In some embodiments, if the calibration image data obtained from each camera does not capture the entirety of the calibration target, resulting in less than all of the features being visible in the image, it may be difficult or even impossible to find the world coordinate matches of the features to be determined, in particular if the calibration target is imaged from several different camera positions and / or orientations. For example, again in terms of 5 if at least one of the first and second images 350 , 352 fails to include the entirety of the checkerboard calibration target (e.g. at least one of the four outer corners, at least one of the four outer edges or portions thereof, omits at least one of the black and second squares or portions thereof) so that only a subset of the checkerboard vertices are visible, the correspondences between vertices in the image and vertices on the target may be ambiguous due to the repeating checkerboard pattern. Although in some examples this problem can be avoided by reducing the size of the calibration target or increasing the distance between the camera and the calibration target to ensure that all of the feature points are within each camera's field of view, these approaches can reduce calibration accuracy.

Different embodiments described herein provide improved calibration targets that allow camera calibration parameters to be determined even when the image data does not include the entirety of the calibration target and calibration target features. In some embodiments, these calibration targets have one or more reference marks that lie at defined locations. The one or more reference markings can each be uniquely identifiable in image data of the calibration target, even if the image data depicts the calibration target in different positions and / or orientations. Thus, even if only a subset of the features are visible in the image data, the image world coordinate matches for the features of the calibration target can be determined by using the fiducial marks as spatial references to unambiguously identify each feature. Advantageously, this approach allows an image to be used for camera calibration as long as it includes all of the reference marks and at least some of the calibration features, thereby improving the flexibility and robustness of the calibration process. This can be particularly advantageous for devices that use multiple cameras, e.g., a UAV implementing a stereo vision capture system, as it can be relatively difficult to arrange the cameras so that the entirety of the calibration target is in the field of view of each camera.

The characteristics of the reference marks can be varied as desired, including shape, size and / or color. Exemplary reference markings contain geometric shapes (for example point, line, circle, triangle, square), symbols or characters (for example alphanumeric characters) or suitable combinations thereof. The calibration targets provided herein can have any suitable number of reference marks, such as at least one, two, three, four, five, or more reference marks. In some embodiments, the calibration targets only have three reference marks. In some embodiments, the calibration targets described herein have only a single type of reference mark. Alternatively, a calibration target can have several different types of reference marks, such as at least two, three, four, five or more different types.

The characteristics of the fiducial marks can be selected to facilitate identification of the fiducial marks from image data, e.g., using image recognition algorithms, feature extraction algorithms, or other computer vision techniques. Optionally, the reference markings can be visually distinguishable from the features in order to facilitate precise identification of the reference marks and features. As another example, the features may be arranged in a repeating pattern (e.g., across a 1D line or a 2D surface) while the fiducial marks are not repeating.

The reference marks can each be positioned at a different location on the calibration target. In some embodiments, the reference marks lie in a defined arrangement relative to the features of the calibration target, so that the spatial relationship (e.g. relative position, orientation) between each feature and at least one of the reference marks is known. For example, one or more features of the calibration target can be overlaid or interspersed with the reference markings. As another example, the fiducial marks can be positioned near or adjacent to one or more features. The reference marks can be positioned on the calibration target so that all of the reference marks and at least some of the features are visible at the same time in images of the calibration target.

Various techniques can be used to make each reference mark uniquely identifiable in image data of the calibration target, even if the image data represent a rotated and / or shifted view of the calibration target. For example, each reference mark can be a different size, shape, and / or color, or can have unique characteristics or markings that enable it to be visually distinguished from the other markings. Alternatively, or in combination, the reference markings can be arranged in a manner that enables each reference mark to be uniquely identified based on their relative positioning. In such embodiments, the reference markings may be similar or identical (e.g., in terms of shape, size, color, or other characteristics). For example, each fiducial mark can be positioned to form vertices of an irregular polygon (e.g., isosceles triangle, unequal-sided triangle). At least some of the angles of an irregular polygon can be different sizes than other angles. Similarly, at least some of the edges of an irregular polygon can have different lengths than other edges. Optionally, all of the angles can be of different sizes and all of the edges can be of different lengths. As an example, in embodiments where the features of the calibration target are arranged in a plane (e.g., a repeating 2D pattern of features such as a grid), the fiducial marks may form a shape that is rotationally symmetrical in the plane.

6th and 7th illustrate fiducial marks that are uniquely identifiable based on relative positioning, in accordance with embodiments. 6th illustrates a first view 400 a plurality of reference marks A, B, C, depicted herein as black circles. The reference markings A, B, C can be arranged relative to one another in order to form vertices of a non-equilateral triangle. 7th illustrates a second view 410 showing a rotation and translation of the first view 400 represents. Due to the asymmetrical nature of the triangle, the reference marks A ', B' and C can be seen in the second view 410 clearly than the reference marks A, B and C respectively in the first view 400 identified accordingly. In some embodiments, identification of corresponding reference marks is based on the positioning of the reference marks relative to the features of the triangle (eg edges, angles) in the first and second views 400 , 410 accomplished. For example, in the embodiment of 6th and 7th , marks B and B 'are both positioned opposite the longest edge of the triangle and marks C and C are both positioned opposite the shortest edge of the triangle. As another example, the markings B and B 'are both positioned at the apex of the largest interior angle of the triangle and the markings C and C are both positioned at the vertex of the smallest interior angle of the triangle. Although the reference marks in 6th and 7th are depicted to form a triangle, it should be appreciated that the reference marks can be arranged to form other irregular polygons (e.g. irregular polygons with three, four, five, six or more vertices). This approach can make each marker uniquely identifiable, regardless of the position and / or orientation from which the reference markers are viewed.

8th illustrates a calibration target 500 , the reference marks 502a , 502b , 502c in accordance with embodiments. Similar to the calibration target 300 assigns the calibration target 500 a chessboard that has a repeating pattern of alternating black and white squares. In the embodiment of 8th are the reference marks 502a-c identical black circles, each positioned within a different white square of the chessboard. However, it should be appreciated that, in alternative embodiments, at least some of the reference marks 502a-c can be white circles positioned within black squares. The reference marks 502a-c lie on the calibration target 500 in a known position relative to the chessboard, e.g. the row and column position of each marker is known. Any reference mark 502a-c can be at a different position on the chessboard (e.g. different row and / or column positions). In some embodiments, the reference marks are 502a-c on the target 500 arranged to form vertices of an irregular triangle. Accordingly, if a picture of the target 500 all three reference marks 502a-c is any reference mark 502a-c clearly identifiable, even if only some of the target 500 is visible (e.g. at least one of the four outer corners, one of the four outer edges, one of the black squares and / or white squares of the chessboard is omitted from the picture). As soon as each reference mark 502a-c was identified in the image data, the spatial relationship (for example rotation, transformation) between the reference markings in the image and the reference markings on the calibration target 500 be determined. This spatial relationship can then be used to determine the world coordinates of each feature (eg corner point) that is mapped in the image data.

9 illustrates a procedure 600 for calibrating one or more imaging devices using a calibration target with one or more reference marks, in accordance with embodiments. At least some of the steps in the process 600 can be implemented by a suitable computing system or device, such as one or more processors. In embodiments where the imaging devices are carried by a moving object, some or all of the steps of the method can be performed 600 carried out when the moving object is not in operation (offline calibration). Alternatively, some or all of the steps can be performed while the moving object is operating (on-line calibration).

In step 610 a plurality of images of a calibration target is received from each of one or more imaging devices (e.g., one or more cameras). The calibration target may have a variety of features and one or more reference marks, as previously described herein. The plurality of features can be arranged in any suitable configuration, such as a repeating pattern such as a grid. The one or more reference marks can be designed to be uniquely identifiable in the images, as discussed previously. The plurality of images provided by each imaging device may include images captured at a variety of different positions and / or orientations relative to the calibration target. Each image can capture all of the fiducial marks and at least some of the plurality of features of the calibration target. In some embodiments, at least one of the images generated by the imaging devices captures less than all of the plurality of features of the calibration target.

In step 620 For example, features and the one or more fiducial marks are identified in each image of the plurality of images provided by each imaging device. For example, each image can be processed using various computer vision algorithms to identify the fiducial marks and features visible in the image. In some embodiments, the algorithm can be carried out based on information relating to the known characteristics of the reference markings and / or features (eg shape, size, color, relative positioning) in order to facilitate identification. Once the reference marks and / or features have been identified, they can be assigned image coordinates that represent their pixel positions within the image. For example, the image coordinates for a circular reference mark can be the coordinates of the center of the circle. Optionally, the images can be corrected and / or in advance before the identification procedure processed to improve identification accuracy, e.g. by reducing noise, discarding images that are judged to be unsuitable for analysis, and so on.

10 until 13th illustrate identification of features and fiducial marks of a calibration target in an image, in accordance with embodiments. 10 Figure 11 illustrates an image capturing a portion of a checkerboard calibration target with three black circular fiducial marks, similar to the target 500 . In particular, Image omits certain portions of the calibration target, such as the top and bottom left corner of the chessboard and the left edge of the chessboard, so that only some of the corner points of the chessboard are visible. However, all three reference marks and the corner points near the reference marks are visible. Since the three reference marks are arranged on the calibration target to form an irregular polygon, the reference marks can be clearly identified even if some areas of the calibration target are not shown in the image. 11th illustrates the picture of 10 following processing to identify the corner points of the checkerboard squares. The corner points can serve as features for camera calibration, as described hereinbefore. In some embodiments, if the vertex identification algorithm fails (eg, no vertices are identified), the image is considered invalid. 12th illustrates the picture of 10 following processing to identify the reference marks. The circular reference marks can be identified using suitable detection methods, such as a Hough transform. If the algorithm fails to identify any reference marks, the image can be considered invalid.

In some embodiments, the fiducial marker algorithm can generate false positives, eg, identify areas of the image that are not fiducial marks. For example, the picture of 12th has some erroneous results where portions of the image that are not the fiducial marks are mistakenly identified. This can occur, for example, when the detection algorithm depends on shape recognition (eg recognition of circles) in order to identify the reference markings, and similarly shaped objects are visible in the image. 13th illustrates filtering the results of 12th to reduce the occurrence of false positives. In embodiments where the fiducial marks are interspersed or overlaid with the features of the calibration target, the image data can be filtered to limit the fiducial identification results to those within the convex envelope of the identified features. In 13th the false positives that lie outside the convex hull of the identified vertices have been removed. The reference markings can be determined from the remaining results based on the knowledge of the relative positioning of the reference markings. In some embodiments, the position information (eg row and column information and / or image coordinates) can be determined for each result. The set of positional information can then be analyzed to identify the fiducial marks that are in correct spatial relationship with one another. For example, again in terms of 13th Since it is known that the reference marks on the target form the vertices of a triangle and the properties of the triangle are given (e.g. edge length, angle), the results can be searched for in order to identify the set that fulfills this geometric specification. If no such results are found, or if more than one set of results are found, the image may be considered invalid.

In step 630 a spatial relationship between the one or more reference marks in each image of the plurality of images and the one or more reference marks on the calibration target is determined. As described above, the reference markings are designed so that the spatial relationship (e.g. translation, rotation, mirroring) can be clearly identified by comparing how the position of the reference markings in the image data differs from the known positions of the reference markings on the actual calibration target differs. Such differences can arise based on the position and / or orientation of the imaging device relative to the calibration target during image acquisition. For example, in embodiments where the one or more fiducial marks form vertices of an irregular polygon in the image data, the spatial relationship can be determined by identifying how the vertices of the irregular polygon in the image data correspond to the known positions of the vertices on the actual target . In embodiments where a checkerboard calibration target is used and the fiducial marks are positioned in certain squares of the chessboard, the spatial relationship can be determined by identifying positional information (e.g., row and column position) for each of the fiducial marks in the image and this information with the known Position information (e.g. row and column position) can be compared on the actual chessboard.

In step 640 a correspondence between image coordinates of features in the image data and global coordinates (world coordinates) for corresponding features on the calibration target is formulated based on the spatial relationship, which is established in step 630 was determined. For example, the positions of each feature point on the actual calibration target can be determined using the determined spatial relationship. Each feature can then be assigned a set of world coordinates indicating the position of the feature point relative to the world coordinate system. Accordingly, since the image coordinates and the world coordinates of each feature are known, the correspondence or association between the image and world coordinates can be formulated.

In embodiments where a plurality of imaging devices are being calibrated, each imaging device that provides a respective set of images may be performed with the step 640 involve identifying the features present in each of the respective image sets and then determining world coordinates for only those features. For example, for binocular calibration using corner points of a chessboard, the step 640 involve identifying the corner points that are visible in both the left and right camera images, and determining the world coordinates of these common corner points. Since both the left and right images depict all of the reference marks of the calibration target as discussed previously, both the left and right images should have at least the features near the reference marks. In alternative embodiments, this process can be reversed by determining world coordinates for all of the features visible in the images generated by multiple imaging devices before determining which features the different image sets have in common.

In step 650 calibration parameters for the one or more imaging devices are determined based on the correspondence that is determined in step 640 was formulated. As previously described herein, the calibration parameters can have parameters that relate to intrinsic camera matrix (e.g. the focal length, position of the image center, pixel size), rotation matrices and translation vectors between the camera and world coordinate systems and / or optical distortion (e.g. radial distortion, eccentric distortion, thin prismatic distortion) for each of the one or more imaging devices. Once the image world coordinate matches are determined for these features, as before with regard to step 640 this correspondence information can be used to solve the various calibration parameters.

wo R=[r₁ r₂ r₃] (eine 3x3 Matrix) ist, T ein 3x1 Vektor ist und λ=Z_c (die Z-Achsenkomponente eines Punkts, nachdem er von dem Weltkoordinatensystem auf das Kamerakoordinatensystem transformiert ist) ist.Various methods can be used to solve calibration parameters. In some embodiments, an initial estimate of the calibration parameters is first obtained. For example, an assignment model (e.g. the assignment model 200 or 250 ) can be used to obtain the following equation between the world coordinate system and the image coordinate system for monocular camera calibration regardless of distortion effects: $$\mathrm{\lambda }\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathrm{M.}\left[\begin{array}{ccc}\mathrm{R.}& ⋮& T\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

where R = [r ₁ r ₂ r ₃ ] (a 3x3 matrix), T is a 3x1 vector, and λ = Z _c (the Z-axis component of a point after it is transformed from the world coordinate system to the camera coordinate system).

In embodiments where the calibration target is planar, such as a planar checkerboard, the target plane can be assumed to be on the X _w , Y _w plane of the world coordinate system _{such that the Z w} coordinate of each feature on the target is zero. Accordingly, the previous equation can be rewritten as: $$\mathrm{\lambda }\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathrm{M.}\left[\begin{array}{cccc}{r}_1& r_2& r_3& T\end{array}\right]\left[\begin{array}{c}{X}_{\mathrm{<figure-callout\; id="0"\; label="w\; Y\; w"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">w</figure-callout>}}& \\ Y_w{}_{}\\ 0\\ 1\end{array}\right]=\mathrm{M.}\left[\begin{array}{ccc}{r}_1& r_2& T\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]$$

The homography matrix H can then be defined as H = [h1 h2 h3] = sM [r1 r2 T], where s = 1 / λ. λ may correspond to an average rather than a single point in the homography matrix equation.

Since the world coordinates of the features are known, e.g. as they are in step 640 have been determined, the homography matrix H can be solved. Based on the definition of the homography matrix, the following equation can be obtained: $$\{\begin{array}{c}{r}_1=\mathrm{\lambda }{\mathrm{M.}}^{-1}{H}_1\\ r_2=\mathrm{\lambda }{\mathrm{M.}}^{-1}{H}_2\\ T=\mathrm{\lambda }{\mathrm{M.}}^{-1}{H}_3\end{array}$$

Since r ₁ and r ₂ are vectors in the rotation matrix, the following equation can be obtained: $$\begin{array}{l}{r}_1^T{r}_2=0\Rightarrow H_1^T{\mathrm{M.}}^{-T}{\mathrm{M.}}^{-1}{H}_2=0\\ r_1^T{r}_2=r_2^T{r}_2=1\Rightarrow H_1^T{\mathrm{M.}}^{-T}{\mathrm{M.}}^{-1}{H}_1=H_1^T{\mathrm{M.}}^{-T}{\mathrm{M.}}^{-1}{H}_2\end{array}$$

definiert wird, kann das Folgende erhalten werden: $$h_j^T=B{h}_j=v_{ji}^{T}b={\left[\begin{array}{c}{h}_{i1}{h}_{j1}\\ h_{i1}{h}_{j2}+h_{i2}{h}_{j1}\\ h_{i1}{h}_{j3}+h_{i3}{h}_{j1}\\ h_{i2}{h}_{j2}\\ h_{i3}{h}_{j2}+h_{i2}{h}_{j3}\\ h_{i3}{h}_{j3}\end{array}\right]}^T\left[\begin{array}{c}{B}_{11}\\ B_{12}\\ B_{13}\\ B_{22}\\ B_{23}\\ B_{33}\end{array}\right]$$

By doing $\mathrm{B.}={\mathrm{M.}}^{-T}{\mathrm{M.}}^{-1}=\left[\begin{array}{ccc}{\mathrm{B.}}_{\mathrm{11th}}& {\mathrm{B.}}_{\mathrm{12th}}& {\mathrm{B.}}_{\mathrm{13th}}\\ {\mathrm{B.}}_{21}& {\mathrm{B.}}_{\mathrm{22nd}}& {\mathrm{B.}}_{23}\\ {\mathrm{B.}}_{31}& {\mathrm{B.}}_{32}& {\mathrm{B.}}_{33}\end{array}\right]$

is defined, the following can be obtained: $$H_j^T=\mathrm{B.}{H}_j=v_{ji}^{T}b={\left[\begin{array}{c}{H}_{i1}{H}_{j1}\\ H_{i1}{H}_{j2}+H_{i2}{H}_{j1}\\ H_{i1}{H}_{j3}+H_{i3}{H}_{j1}\\ H_{i2}{H}_{j2}\\ H_{i3}{H}_{j2}+H_{i2}{H}_{j3}\\ H_{i3}{H}_{j3}\end{array}\right]}^T\left[\begin{array}{c}{\mathrm{B.}}_{\mathrm{11th}}\\ {\mathrm{B.}}_{\mathrm{12th}}\\ {\mathrm{B.}}_{\mathrm{13th}}\\ {\mathrm{B.}}_{\mathrm{22nd}}\\ {\mathrm{B.}}_{23}\\ {\mathrm{B.}}_{33}\end{array}\right]$$

According to equation (8), the following can be obtained: $$Vb=\left[\begin{array}{c}{V}_{\mathrm{12th}}^T\\ V_{\mathrm{11th}}^T-V_{\mathrm{22nd}}^T\end{array}\right]b=0$$

In equation (10), V is a 2x6 matrix and is transformed from the homography matrix H. To solve b, at least three different H are used to narrow the matrix B. Accordingly, the parameter estimation can be carried out with data from at least three different images of the calibration target at different positions and / or orientations.

If matrix B can be solved, matrix M can also be solved. Subsequently, R and T can be solved according to equation (7). The closed solution, based on different forms of the matrix M, can be as follows:

If the "tangential effect" is not taken into account $$\begin{array}{l}\mathrm{M.}=\left[\begin{array}{ccc}{f}_x& 0& {\mathrm{<figure-callout\; id="0"\; label="u"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">u</figure-callout>}}_0\\ 0& {\mathrm{<figure-callout\; id="0"\; label="f\; y\; v"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">f</figure-callout>}}_y& v_{}{}_0\\ 0& 0& 1\end{array}\right]\text{is}\text{, then}\\ \{\begin{array}{l}{ƒ}_x=\sqrt{k/{\mathrm{B.}}_{\mathrm{11th}}}\hfill \\ {ƒ}_y=\sqrt{k{\mathrm{B.}}_{\mathrm{11th}}/{\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{\mathrm{12th}}-{\mathrm{B.}}_{\mathrm{12th}}^2}\hfill \\ {\mathrm{<figure-callout\; id="0"\; label="u"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">u</figure-callout>}}_0=-{\mathrm{B.}}_{\mathrm{13th}}{f}_x^2/\mathrm{<figure-callout\; id="0"\; label="k\; v"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">k</figure-callout>}\hfill & \\ v_{}\hfill {}_0=\left({\mathrm{B.}}_{\mathrm{12th}}{\mathrm{B.}}_{\mathrm{13th}}-{\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{23}\right)/\left({\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{\mathrm{22nd}}-{\mathrm{B.}}_{\mathrm{12th}}^2\right)\hfill \\ k={\mathrm{B.}}_{33}-{\mathrm{B.}}_{\mathrm{13th}}^2+v_0\left({\mathrm{B.}}_{\mathrm{12th}}{\mathrm{B.}}_{\mathrm{13th}}-{\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{23}\right)/{\mathrm{B.}}_{\mathrm{11th}}\hfill \end{array}\end{array}$$

If the "tangential effect" is not taken into account $$\begin{array}{l}\mathrm{M.}=\left[\begin{array}{cc}{f}_{\mathrm{<figure-callout\; id="0"\; label="x\; \gamma \; u"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">x</figure-callout>}}& \gamma & u_{}{}_0\\ 0& {\mathrm{<figure-callout\; id="0"\; label="f\; y\; v"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">f</figure-callout>}}_y& v_{}{}_0\\ 0& 0& 1\end{array}\right]\text{then}\\ \{\begin{array}{l}{ƒ}_x=\sqrt{k/{\mathrm{B.}}_{\mathrm{11th}}}\hfill \\ {ƒ}_y=\sqrt{k{\mathrm{B.}}_{\mathrm{11th}}/{\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{\mathrm{12th}}-{\mathrm{B.}}_{\mathrm{12th}}^2}\hfill \\ \gamma =-{\mathrm{B.}}_{\mathrm{12th}}{f}_x^2{f}_y/\mathrm{<figure-callout\; id="0"\; label="k\; u"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">k</figure-callout>}\hfill & \\ u_{}\hfill {}_0=\mathrm{<figure-callout\; id="0"\; label="\gamma \; v"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">\gamma </figure-callout>}{v}_{}{}_0/f_y-{\mathrm{B.}}_{\mathrm{13th}}{f}_x^2/\mathrm{<figure-callout\; id="0"\; label="k\; v"\; filenames="DE202014011514U1\_0041.png"\; state="\{\{state\}\}">k</figure-callout>}\hfill & \\ v_{}\hfill {}_0=\left({\mathrm{B.}}_{\mathrm{12th}}{\mathrm{B.}}_{\mathrm{13th}}-{\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{23}\right)/\left({\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{\mathrm{22nd}}-{\mathrm{B.}}_{\mathrm{12th}}^2\right)\hfill \\ k={\mathrm{B.}}_{33}-\left[{\mathrm{B.}}_{\mathrm{13th}}^2+v_0\left({\mathrm{B.}}_{\mathrm{12th}}{\mathrm{B.}}_{\mathrm{13th}}-{\mathrm{B.}}_{\mathrm{11th}}{\mathrm{B.}}_{23}\right)\right]/{\mathrm{B.}}_{\mathrm{11th}}\hfill \end{array}\end{array}$$

After the initial solution has been obtained, the calibration parameters can be optimized, for example using the Levenberg-Marquardt algorithm. In some embodiments, the optimal solution minimizes the following object function: $${\displaystyle \sum {}_i{\displaystyle \sum {}_j}}{\Vert m_{ij}-\widehat{m}\left(\mathrm{M.}\text{,}K\text{,}\mathrm{P.}\text{,}\mathrm{R.}\text{,}T\right)\Vert }^2$$

wo R die Rotationsmatrix von dem rechten Kamerakoordinatensystem zu dem linken Kamerakoordinatensystem ist und T der Translationsvektor von dem linken Kamerakoordinatensystemursprung O₁ zu dem rechten Kamerakoordinatensystemursprung O_r ist.In embodiments where multiple imaging devices are being calibrated (eg, two or more cameras), the step 650 also include solving for the rotation matrix R and the translation vector T between each of the camera coordinate systems. These extrinsic parameters can be obtained based on the monocular calibration methods described herein. For example, in binocular calibration (calibration of two imaging devices) for a spatial point observed by a left and right camera, the coordinate P ₁ in the left camera coordinate system and the coordinate P _r in the right camera coordinate system can satisfy the following relationship: $${\text{P.}}_{\text{1}}={\text{RP}}_{\text{r}}+\text{T}$$

where R is the rotation matrix from the right camera coordinate system to the left camera coordinate system and T is the translation _{vector from the left camera coordinate system origin O 1} to the right camera coordinate system origin O r.

14th illustrates a model 800 spatial relationships between coordinate systems in binocular camera calibration, in accordance with embodiments. The world coordinate system (X _w , Y _w , Z _w with origin O _w ) relates to the left camera _{coordinate system (X 1} , Y ₁ , Z ₁ with origin O ₁ ) through rotation matrix R ₁ and translation _{vector T 1} and to the right camera coordinate system ( X _r , Y _r , Z _r with origin O _r ) by rotation matrix R _r and translation _vector T r. These rotation matrices and translation vectors can be obtained using the methods previously described herein. the Rotation matrix R and translation vector T for converting between the left and right camera coordinate systems can then be determined by the following relationship: $$\{\begin{array}{c}\mathrm{R.}={\mathrm{R.}}_l{\left({\mathrm{R.}}_r\right)}^T\\ T=T_l-\mathrm{R.}{T}_r\end{array}$$

Once the calibration parameters have been determined in accordance with the methods described herein, the parameters can be used to process image data obtained from the imaging devices so that the image data can be made suitable for various downstream applications, e.g. Map creation, object recognition, navigation, etc. For example, based on the determined calibration parameters, the image data can be processed to correct for the effects of optical distortion associated with the one or more imaging devices, also referred to herein as "rectification" will. As another example, in embodiments where two or more imaging devices are used to capture respective sets of the image data, the calibration parameters can be used to perform rectification, e.g., to enable the image data sets to be used for stereo vision.

15th illustrates a model 900 an association between distorted image coordinates and rectified image coordinates, in accordance with embodiments. Distortion can occur in images captured by an imaging device due to the optical properties of the lens. Image rectification can be used to remove the distortion effects in an image captured by the imaging device (e.g. an image that has distorted image or pixel coordinates (u ₁ , v ₁ ) corresponding to the distorted image plane _{coordinates (X d} , Y _d )) in order to obtain a rectified image according to the ideal pinhole camera model (eg an image which has rectified picture or pixel coordinates (u ₂ , v ₂ ) corresponding to rectified picture plane _{coordinates (X p} , Y _p )). Accordingly, image rectification can be used to transform a distorted image into a rectified image. The image rectification procedure can be carried out based on previously determined intrinsic parameters for the imaging devices (for example distortion parameters, intrinsic camera matrix M). Such parameters can be determined using the approaches presented herein.

Various methods can be used to perform image rectification. For example, the rectified image plane _{coordinates (X p} , Y _p ) can be determined from the rectified pixel coordinates (u ₂ , v ₂ ) according to the following equation: $$\left[\begin{array}{c}{X}_p\\ Y_p\\ 1\end{array}\right]={\mathrm{M.}}^{-1}\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]$$

The distorted image plane coordinates (X _d , Y _d ) can be derived from the rectified image plane coordinates (X _p , Y _p ) based on a mathematical distortion model (e.g. that defined by the distortion equation ( 3 ), which was previously presented herein, described model) and previously determined values for the distortion parameters can be determined. The pixel coordinates of the distorted image (u ₁ , v ₁ ) can then be obtained by the following equation: $$\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=\mathrm{M.}\left[\begin{array}{c}{X}_d\\ Y_d\\ 1\end{array}\right]$$

The gray scale value of the rectified pixel coordinates (u ₂ , v ₂ ) can be obtained by interpolating the distorted pixel coordinates (u ₁ , v ₁ ). The interpolation can be performed using any suitable technique, such as nearest neighbor interpolation or bilinear interpolation. Accordingly, the rectified image can be obtained.

Similar to the image rectification process described herein, image rectification, also known as stereo rectification, may involve an image-to-image transformation process. However, image rectification does involve also a spatial transformation on the camera coordinate system level. Image rectification can be used to process image data captured by multiple imaging devices to facilitate the implementation of stereo vision algorithms, such as creating depth maps, disparity maps, and other 3D representations of a scene. For example, in some embodiments, stereo vision algorithms may involve point matches between images captured by different cameras (e.g., a left camera and a right camera). Image rectification can be used to project the image data obtained from each camera onto a common plane, thereby simplifying the procedure for searching for related pixels in the images.

16 and 17th illustrate an image rectification procedure for a binocular camera system, in accordance with embodiments. 16 illustrates a model 1000 an assignment between the left camera coordinate system (camera 1) and the right camera coordinate system (camera 2) for stereo vision. The image rectification process may involve determining the rotation matrices R ₁ , R _r, and R _rect that are used to rectify the left and right image data. In some embodiments, these matrices are designed to maximize the overlapping portions of the left and right images after rectification and to minimize the amount of image deformation. Furthermore, if the same matrix M is used in both rectification images, the rectified images can have the same scale, thereby making it easier to calculate the depth field.

wo R die Rotationsmatrix von dem rechten Kamerakoordinatensystem (X_r, Y_r, Z_r) zu dem linken Kamerakoordinatensystem (X₁, Y₁, Z₁) ist. R₁ und R_r sind die Rotationsmatrizen von dem linken Kamerakoordinatensystem zu einem ersten vorübergehenden Koordinatensystem (X_{1_temp}, Y_{1_temp}, Z_{1_temp}) beziehungsweise dem rechten Kamerakoordinatensystem zu einem zweiten vorübergehenden Koordinatensystem (X_{r_temp,} Y_{r_temp}, Z_{r_temp}) . In manchen Ausführungsformen kann $R_l^T$

gleich R_r gesetzt sein, sodass der Rotationswinkel der Rotationsmatrix R gleichmäßig geteilt wird. Das linke und rechte Kamerakoordinatensystem können gemäß R₁ beziehungsweise R_r transformiert werden, sodass die räumliche Lage der zwei Kamerakoordinatensysteme nach Transformation identisch ist. Jedoch kann es einen Translationsvektor T_temp=R₁T in dem Koordinatensystem geben, der erhalten wird, indem das linke Kamerakoordinatensystem gemäß R₁ transformiert wird. 17th illustrates a model 1050 an assignment between unrectified and rectified camera coordinate systems. In some embodiments, the rotation matrices R ₁ and R _r can satisfy the equation $${\mathrm{R.}}_l^{\mathrm{R.}}{\mathrm{R.}}_r=\mathrm{R.}$$

where R is the rotation matrix from the right camera coordinate system (X _r , Y _r , Z _r ) to the left camera coordinate system (X ₁ , Y ₁ , Z ₁ ). R ₁ and R _r are the rotation matrices from the left camera coordinate system to a first temporary coordinate system (X _{1_temp} , Y _{1_temp} , Z _{1_temp} ) and the right camera coordinate system to a second temporary coordinate system (X _{r_temp,} Y _{r_temp} , Z _{r_temp} ). In some embodiments, may ${\mathrm{R.}}_l^T$

be set equal to R _r , so that the rotation angle of the rotation matrix R is divided equally. The left and right camera _{coordinate systems can be transformed according to R 1} and R _r, respectively, so that the spatial position of the two camera coordinate systems is identical after transformation. However, there may be a translation _vector T temp = R ₁ T in the coordinate system obtained by transforming the left camera coordinate system according to R ₁ .

wo $e_1=\frac{T_{temp}}{\Vert T_{temp}\Vert }\text{, }{e}_2=\frac{{\left[-T_{temp\text{\_}y}{T}_{temp\text{\_}x}0\right]}^T}{\sqrt{T_{temp\text{\_}x}^2+T_{temp\text{\_y}}^2}}\text{, }{e}_3=e_1\times e_2\text{, }{T}_{temp}=R_{l}T$

(eine 3x1 Matrix) ist, T_{temp_x} als die erste Spalte der ersten Reihe von T_temp definiert ist und T_{temp_y} als die erste Spalte der zweiten Reihe von T_temp definiert ist.The two coordinate systems can then be transformed simultaneously by the rotation matrix R _rect , so that the XOY planes of the two coordinate systems are coplanar and both X axes are parallel to the base line. The matrix R _rect can be determined subjectively. For example, R _rect can be expressed as $${\mathrm{R.}}_{rect}=\left[\begin{array}{c}{e}_1^T\\ e_2^T\\ e_3^T\end{array}\right]$$

Where $e_1=\frac{T_{temp}}{\Vert T_{temp}\Vert }\text{,}{e}_2=\frac{{\left[-T_{temp\text{\_}y}{T}_{temp\text{\_}x}0\right]}^T}{\sqrt{T_{temp\text{\_}x}^2+T_{temp\text{\_y}}^2}}\text{,}{e}_3=e_1\times e_2\text{,}{T}_{temp}={\mathrm{R.}}_{l}T$

(a 3x1 matrix), T _{temp_x is defined} as the first column of the first row of T _temp , and T _{temp_y is defined} as the first column of the second row of T _temp .

In some embodiments, after the rotation of the camera _{coordinate systems, there is only one translation vector T rect} = T _rect R ₁ T, which is parallel to the X axis and between the two coordinate systems. Optionally, the matrix M can be replaced by the matrix M ₁ (see 10 ).

Dies kann skaliert werden, um die folgende Beziehung zu erhalten: $$\begin{array}{l}{X}_{pl}^{\text{'}}=X_{pl}/Z_{pl}\\ Y_{pl}^{\text{'}}=Y_{pl}/Z_{pl}\end{array}$$

After the rotation matrices R ₁ , R _r and R _rect and the assignment matrix M have been determined, the image rectification procedure can be carried out in a manner similar to the individual image rectification procedure described above. Image rectification can be performed separately on each set of images obtained from each individual imaging device. For example, again in terms of 16 a reverse assignment can be carried out for a point (u ₃ , v ₃ ) in order to obtain the coordinate ((X _p1 , Y _p1 , Z _p1 ) under the original left camera coordinate system: $$\left[\begin{array}{c}{X}_{pl}\\ Y_{pl}\\ Z_{pl}\end{array}\right]={\left({\mathrm{R.}}_{rect}{\mathrm{R.}}_l\mathrm{M.}\right)}^{-1}\left[\begin{array}{c}{u}_3\\ v_3\\ 1\end{array}\right]$$

This can be scaled to have the following relationship: $$\begin{array}{l}{X}_{pl}^{\text{'}}=X_{pl}/Z_{pl}\\ Y_{pl}^{\text{'}}=Y_{pl}/Z_{pl}\end{array}$$

The distorted coordinates can be determined according to the previously determined distortion parameters and a distortion model (eg equation (3)). The pixel coordinates (u ₁ , v ₁ ) under the original left coordinate system can then be obtained: $$\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=\mathrm{M.}\left[\begin{array}{c}{X}_d\\ Y_d\\ 1\end{array}\right]$$

The gray scale value of the rectified image coordinates (u ₃ , v ₃ ) can be determined using any suitable interpolation algorithm. This rectification process can also be used to obtain rectified image coordinates for the right camera.

18th Figure 3 illustrates binocular camera images following processing to perform image rectification, in accordance with embodiments. The images show a first rectified image 1100 obtained from a left camera and a second rectified image 1102 obtained from a right camera. In particular, there are objects in the first image 1100 horizontally with corresponding objects in the second image 1102 aligned as from the superimposed horizontal lines 1104 is specified.

Once the image data obtained from the one or more images has been processed by rectification and / or rectification as described herein, the images can be used for various applications, such as computer vision applications. In some embodiments, the images can be used to restore spatial information for the imaged scene, such as the distances of objects in the scene from the imaging devices. This spatial information can be used, for example, to generate 3D representations of the environment, such as depth maps, disparity maps, occupancy grids, point clouds and so on.

19th illustrates a procedure 1200 for generating depth maps from image data, in accordance with embodiments. At least some of the steps in the process 1200 can be implemented by a suitable computing system or device, such as one or more processors.

In step 1210 one or more images are obtained from each of one or more imaging devices. The imaging devices can be configured to capture the images at the same time or substantially at the same time in order to generate corresponding images of a scene. The imaging devices may be relative to one another to create images of the same scene from different positions and / or orientations. For example, a UAV equipped with a binocular vision system may have a left camera and a right camera that capture images of the environment surrounding the UAV at the same time but from two different positions and / or orientations. Accordingly, objects in the scene will have different image coordinates in images obtained from different imaging devices.

In step 1220 For example, the one or more images from each of the one or more imaging devices are processed based on calibration parameters for the one or more imaging devices to generate a 3D depth map. The calibration parameters can include intrinsic and / or extrinsic parameters for the imaging devices, as previously described herein, and can be previously described using one of the calibration methods described herein, such as the method 600 , be determined. The calibration process can be an online calibration process or an offline calibration process. In some embodiments, the calibration parameters are determined using a plurality of calibration images of a calibration target from each of the one or more imaging devices. The calibration target can be any of the embodiments illustrated herein, such as a checkerboard calibration target. For example, the calibration target can have a plurality of features arranged in a repeating pattern and have one or more reference marks. The one or more reference markings can be designed to be uniquely identifiable in each case within each calibration image that is obtained from imaging devices, for example as before with regard to the embodiments of FIG 6th , 7th and 8th was discussed. In some embodiments, at least one of the calibration images captures less than all of the features of the calibration target. As previously discussed, the methods described herein can be used to determine calibration parameters from the calibration images even if some or all of the images do not capture all of the features of the calibration target features, as described herein.

Various methods can be used to process the images to enable generation of a 3D depth map from the images. For example, the images can be processed by the rectification and / or rectification techniques discussed herein. The rectified images can then be used as input into various stereo vision algorithms to restore depth information for the objects in the images, for which purpose a depth map of the imaged scene is generated. For example, corresponding images (e.g., images captured at the same time or substantially the same time) obtained from different imaging devices can be compared with one another to determine the disparities between the images. Based on the disparity information and information regarding the spatial relationships between the imaging devices, the distance between the object and the imaging devices can be determined. For example, the disparity between the image coordinates of the same object in images of different imaging devices can be calculated. The distance from the imaging devices to the object can be expressed in terms of disparity, the focal lengths of the imaging devices, and the spatial relationship between the imaging devices (e.g., distance between the imaging devices). Accordingly, once the focal length and spatial relationship information has been determined (e.g., using the calibration procedures described herein), the distance can be calculated.

This process can be repeated to generate a 3D depth map for all points within the imaged scene. Such depth maps can be used in a wide variety of applications. For example, in embodiments where the imaging devices are carried by a UAV, the depth map generated based on the image data from the imaging devices can be used to provide information indicative of the distances of various objects from the UAV. This information can be used to facilitate various UAV functionalities such as obstacle avoidance, navigation, target tracking and the like.

The systems, devices, and methods described herein can be applied to a wide variety of moving objects. As previously mentioned, any description of an aircraft herein can be applied to and used for any moving object. A moving object of the present invention can be configured to move within any suitable environment, such as in air (e.g., a fixed wing aircraft, a rotary wing aircraft, or an aircraft that has neither rigid wings nor rotating wings), in water (e.g. a ship or a a submarine), on the ground (e.g. a motor vehicle, such as a car, truck, bus, minibus, motorcycle; a movable structure or frame, such as a pole, fishing rod; or a train), underground (e.g. a subway ), in space (e.g. a space shuttle, satellite, or probe), or any combination of these environments. The moving object can be a vehicle, such as a vehicle described elsewhere herein. In some embodiments, the movable object can be attached to a living subject, such as a human or an animal. Suitable animals can include birds, dogs, cats, horses, cattle, sheep, pigs, dolphins, rodents, or insects.

The moving object may be able to move freely within the environment with respect to sixty degrees of freedom (e.g., thirty degrees of freedom in translation and thirty degrees freedom in rotation). Alternatively, the movement of the moving object may be restricted with respect to one or more degrees of freedom, such as by a predetermined path, track, or orientation. The movement can be actuated by any suitable actuating mechanism, such as a machine or a motor. The actuating mechanism of the moving object can be powered by any suitable source of energy, such as electrical energy, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, atomic energy, or any suitable combination thereof. The moving object can be self propelled via a drive system as described elsewhere herein. The propulsion system can optionally run on an energy source such as electrical energy, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, atomic energy, or any suitable combination thereof. Alternatively, the moving object can be carried by a living being.

In some examples, the moving object can be a vehicle. Suitable vehicles can include watercraft, aircraft, spacecraft, or ground vehicles. For example, aircraft can be fixed wing aircraft (e.g. aircraft, gliders), rotary wing aircraft (e.g. helicopters, rotor aircraft), aircraft that have both rigid wings and rotary wings, or aircraft that do not have either (e.g. small airships, hot air balloons). A vehicle can be self-propelled, such as through the air, on or in water, in space, or self-propelled on or under the ground. A self-propelled vehicle can utilize a propulsion system, such as a propulsion system, that includes one or more machines, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles, or any combination thereof. In some examples, the propulsion system may be used to enable the moving object to take off from a surface, land on a surface, maintain its current position and / or orientation (eg, hover), change orientation, and / or change position .

The moving object can be remotely controlled by a user or controlled locally in or on the moving object by an occupant. In some embodiments, the moving object is an unmanned moving object, such as a UAV. An unmanned moving object, such as a UAV, cannot have an occupant on board the moving object. The moving object can be controlled by a human or an autonomous control system (e.g., a computer control system), or any suitable combination thereof. The moving object can be an autonomous or semi-autonomous robot, such as a robot configured with artificial intelligence.

The moving object can be of any suitable size and / or dimensions. In some embodiments, the movable object may be sized and / or dimensioned to have a human occupant within or on top of the vehicle. Alternatively, the movable object may have a size and / or dimensions that are smaller than those capable of having a human occupant within or on the vehicle. The movable object can have a size and / or dimensions that are suitable for being lifted or carried by a human being. Alternatively, the moving object may be larger than a size and / or dimensions suitable for being lifted or carried by a human. In some examples, the moving object can have a maximum dimension (e.g. length, width, height, diameter, diagonal) less than or equal to approximately: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 have m. The maximum dimension can be greater than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance between shafts of opposing rotors of the moving object may be less than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Alternatively, the distance between shafts of opposing rotors can be greater than or equal to approximately: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 m.

In some embodiments, the moving object can have a volume smaller than 100 cm x 100 cm x 100 cm, smaller than 50 cm x 50 cm x 30 cm or smaller than 5 cm x 5 cm x 3 cm. The total volume of the moving object can be less than or equal to approximately: 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , 200 cm ³ , 300 cm ³ , 500 cm ³ , 750 cm ³ , 1000 cm ³ , 5000 cm ³ , 10,000 cm ³ , 100,000 cm ³ , 1 m ³ or 10 m ³ . Conversely, the total volume of the moving object can be greater than or equal to approximately: 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , 200 cm ³ , 300 cm ³ , 500 cm ³ , 750 cm ³ , 1000 cm ³ , 5000 cm ³ , 10,000 cm ³ , 100,000 cm ³ , 1 m ³ or 10 m ³ .

In some embodiments, the moving object may be less than or equal to a footprint (which refers to the lateral cross-sectional area enclosed by the moving object) for example: 32,000 cm ² , 20,000 cm ² , 10,000 cm ² , 1,000 cm ² , 500 cm ² , 100 cm ² , 50 cm ² , 10 cm ² or 5 cm ² . Conversely, the footprint can be greater than or equal to approximately: 32,000 cm ² , 20,000 cm ² , 10,000 cm ² , 1,000 cm ² , 500 cm ² , 100 cm ² , 50 cm ² , 10 cm ² or 5 cm ² .

In some examples, the moving object cannot weigh more than 1000 kg. The weight of the moving object can be less than or equal to about: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg , 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg , 0.1 kg, 0.05 kg or 0.01 kg. Conversely, the weight can be greater than or equal to approximately: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, Be 0.1 kg, 0.05 kg or 0.01 kg.

In some embodiments, a moving object may be small relative to a load carried by the moving object. The load may comprise a payload and / or a carrier, as will be described in more detail below. In some examples, a ratio of a moving object weight to a load weight may be greater than, less than, or equal to about 1: 1. In some examples, a ratio of a moving object weight to a load weight may be greater than, less than, or equal to about 1: 1. Optionally, a ratio of a carrier weight to a load weight can be greater than, less than, or equal to about 1: 1. If desired, the ratio of a moving object weight to a load weight can be less than or equal to: 1: 2, 1: 3, 1: 4, 1: 5, 1:10 or even less. Conversely, the ratio of a moving object weight to a load weight can also be greater than or equal to: 2: 1, 3: 1, 4: 1, 5: 1, 10: 1 or even higher.

In some embodiments, the moving object can have low power consumption. For example, the moving object may consume less than about: 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h or less. In some examples, a carrier of the moving object can have low energy consumption. For example, the wearer may consume less than about: 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or less. Optionally, a payload of the moving object may have low energy consumption, such as less than about: 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or less.

20th illustrates an unmanned aerial vehicle (UAV) 1300 in accordance with embodiments of the present invention. The UAV can be an example of a moving object as described herein. The UAV 1300 may have a drive system that has four rotors 1302 , 1304 , 1306 and 1308 having. Any number of rotors can be provided (e.g., one, two, three, four, five, six, or more). The rotors can be embodiments of the self-priming rotors described elsewhere herein. The rotors, rotor assemblies, or other propulsion systems of the unmanned aerial vehicle may allow the unmanned aerial vehicle to hover / maintain position, change orientation, and / or change location. The spacing between shafts of opposing rotors can be any suitable length 1310 exhibit. For example, the length 1310 be less than or equal to 2 m or less than or equal to 5 m. In some embodiments, the length can be 1310 be within a range of 40 cm to 1 m, from 10 cm to 2 m or from 5 cm to 5 m. Any description of a UAV herein can be applied to a moving object, such as a moving object of a different type, and vice versa.

In some embodiments, the moving object can be configured to carry a load. The load can include one or more of passengers, cargo, equipment, instruments, and the like. The load can be provided within a housing. The housing can be separate from a housing for the moving object or it can be part of a housing for a moving object. Alternatively, the load may be provided with a housing while the movable object has no housing. Alternatively, portions of the load or the entire load can be provided without a housing. The load can be fixed rigidly relative to the moving object. Optionally, the load can be movable (e.g., slidable or rotatable relative to the movable object) relative to the movable object.

In some embodiments, the load includes a payload. The payload can be configured not to perform any operation or function. Alternatively, the payload may be a payload configured to perform an operation or function, also known as a functional payload. For example, the payload can include one or more sensors to monitor one or more targets. Any suitable sensors can be incorporated into the payload, such as an image pickup device (e.g., a camera), a sound pickup device (e.g., a directional microphone), an infrared imaging device, or an ultraviolet imaging device. The sensor can capture data (e.g. a photograph) or static Provide dynamic acquisition data (e.g. a video). In some embodiments, the sensor provides detection data for the target of the payload. Alternatively or in combination, the payload can have one or more emitters in order to provide signals to one or more destinations. Any suitable emitter can be used, such as a lighting source or a sound source. In some embodiments, the payload includes one or more transceivers, such as to communicate with a module remote from the moving object. Optionally, the payload can be configured to interact with the environment or a target. For example, the payload may include a tool, instrument, or mechanism capable of manipulating objects, such as a robotic arm.

Optionally, the load can have a carrier. The carrier can be provided for the payload and the payload can be coupled to the moving object via the carrier either directly (e.g., directly in contact with the moving object) or indirectly (e.g. not in contact with the moving object). Conversely, the payload can be mounted on the moving object without the need for a carrier. The payload can be formed integrally with the carrier. Alternatively, the payload can be releasably coupled to the carrier. In some embodiments, the payload can include one or more payload elements, and one or more of the payload elements can be movable relative to the movable object and / or the carrier, as previously described.

The carrier can be formed integrally with the moving object. Alternatively, the carrier can be detachably coupled to the movable object. The carrier can be coupled directly or indirectly to the moving object. The carrier can provide support for the payload (e.g., carry at least a portion of the weight of the payload). The carrier may have a suitable mounting structure (e.g. a gimbal platform) capable of stabilizing and / or directing the movement of the payload. In some embodiments, the carrier can be adapted to control the state of the payload (e.g., position and / or orientation) relative to the moving object. For example, the carrier can be configured to move relative to the moving object (e.g. with respect to one, two or three degrees of translation and / or one, two or three degrees of rotation) such that the payload is relative to its position and / or orientation an appropriate frame of reference regardless of the movement of the moving object. The frame of reference can be a fixed frame of reference (e.g. the surrounding environment). Alternatively, the reference frame can be a moving reference frame (e.g. the moving object, a payload target).

In some embodiments, the carrier may be configured to allow movement of the payload relative to the carrier and / or moving object. The movement can be translation with respect to up to three degrees of freedom (e.g., along one, two, or three axes) or rotation with respect to up to three degrees of freedom (e.g., about one, two or three axes), or any suitable combination thereof.

In some examples, the carrier can include a carrier frame assembly and a carrier actuation assembly. The support frame assembly can provide structural support for the payload. The support frame assembly can include individual support frame components, some of which can be movable relative to one another. The carrier actuation assembly may include one or more actuators (e.g., motors) that actuate movement of the individual carrier frame components. The actuators can allow the movement of several carrier frame components at the same time or can be configured to allow the movement of a single carrier frame component at a time. The movement of the carrier frame components can produce a corresponding movement of the payload. For example, the carrier actuation assembly can actuate rotation of one or more carrier frame components about one or more axes of rotation (e.g., roll axis, pitch axis, or yaw axis). The rotation of the one or more support frame components can cause a payload to rotate about one or more axes of rotation relative to the moving object. Alternatively or in combination, the carrier actuation assembly can actuate a translation of one or more carrier frame components along one or more translation axes and thereby generate a translation of the payload along one or more corresponding axes relative to the movable object.

In some embodiments, the movement of the movable object, the carrier and the payload relative to a fixed reference frame (eg the surrounding environment) and / or to one another can be controlled by a terminal. The terminal may be a remote control device at a location remote from the moving object, carrier, and / or payload. The terminal device can be arranged on a support platform or fixed to it. Alternatively, the terminal can be a handheld device or a device that can be worn on the body. For example, the terminal device can have a smartphone, tablet, laptop, computer, glasses, gloves, helmet, microphone or suitable combinations thereof. The terminal device can have a user interface, such as a keyboard, mouse, control lever, touch screen or an ad. Any suitable user input can be used to interact with the terminal, such as manually entered commands, voice control, gesture control or position control (e.g. via a movement, a location or a tilt of the terminal).

The terminal can be used to control any suitable state of the moving object, carrier and / or payload. For example, the terminal can be used to control the position and / or orientation of the movable object, carrier and / or the payload relative to a fixed reference from and / or to one another. In some embodiments, the terminal can be used to control individual elements of the moving object, carrier, and / or the payload, such as the carrier's actuation assembly, a sensor of the payload, or an emitter of the payload. The terminal may have a wireless communication device adapted to communicate with one or more of the moving object, carrier or payload.

The terminal can have a suitable display unit for viewing information about the moving object, carrier and / or the payload. For example, the terminal may be configured to display display information of the moving object, carrier, and / or payload in terms of position, translational speed, translational acceleration, orientation, angular speed, angular acceleration, or any suitable combination thereof. In some embodiments, the terminal may display information provided by the payload, such as data provided by a functional payload (e.g., images recorded by a camera or other image capture device).

Optionally, the same terminal can control both the moving object, the carrier and / or the payload or a state of the moving object, carrier and / or the payload, and also receive and / or display information from the moving object, carrier and / or payload. For example, a terminal device can control the positioning of the payload relative to an environment while image data recorded from the payload or information about the position of the payload is displayed. Alternatively, different terminals can be used for different functions. For example, a first terminal can control movement or a state of the moving object, carrier and / or the payload, while a second terminal can receive and / or display information from the moving object, carrier and / or the payload. For example, a first terminal can be used to control the positioning of the payload relative to an environment, while a second terminal displays image data that has been recorded from the payload. Various communication modes can be used between a moving object and an integrated terminal that both controls the moving object and receives data, or between the moving object and a plurality of terminals that both control the moving object and receive data. For example, at least two different communication modes can be established between the moving object and the terminal which both controls the moving object and receives data from the moving object.

21 illustrates a moving object 1400 who have a carrier 1402 and a payload 1404 in accordance with embodiments. Though the moving object 1400 is depicted as an airplane, this illustration is not intended to be limiting and any suitable type of moving object may be used, as previously described herein. One skilled in the art will appreciate that any of the embodiments described herein can be applied to any suitable moving object (e.g., a UAV) in the context of aircraft systems. In some examples, the payload can 1404 on the moving object 1400 be provided without the carrier 1402 to need. The moving object 1400 can drive mechanisms 1406 , a registration system 1408 and a communication system 1410 exhibit.

The drive mechanisms 1406 may have one or more of rotors, propellers, blades, machines, motors, wheels, axles, magnets or nozzles, as previously described. For example, the drive mechanisms 1406 self-priming rotors, rotor assemblies, or other rotating drive units as disclosed elsewhere herein. The moving object may have one or more, two or more, three or more, or four or more drive mechanisms. The drive mechanisms can all be of the same type. Alternatively, one or more drive mechanisms can be different types of drive mechanisms. The drive mechanisms 1406 can on the moving object 1400 using any suitable means, such as a support member (e.g., drive shaft), as described elsewhere herein. The drive mechanisms 1406 can be at any suitable section of the moving object 1400 such as the top, bottom, front, back, sides, or any suitable combination thereof.

In some embodiments, the drive mechanisms 1406 the moving object 1400 allow vertical take-off from a surface or vertical land on a surface without any horizontal movement of the moving object 1400 to need (e.g. without driving down a taxiway). Optionally, the drive mechanisms 1406 be operational, the moving object 1400 to allow floating in the air at a certain position and / or orientation. One or more of the drive mechanisms 1400 can be controlled independently of the other drive mechanisms. Alternatively, the drive mechanisms 1400 configured to be controlled at the same time. For example, the moving object can 1400 have a plurality of horizontally aligned rotors that can provide stroke and / or feed to the moving object. The multiple horizontally aligned rotors can be actuated to provide vertical takeoff, vertical landing, and hovering capacity to the moving object 1400 provide. In some embodiments, one or more of the horizontally aligned rotors can rotate clockwise while one or more of the horizontal rotors rotate counterclockwise. For example, the number of clockwise rotors may be equal to the number of counterclockwise rotors. The rate of rotation of each of the horizontally aligned rotors can be varied independently in order to control the stroke and / or feed generated by each rotor and thereby the spatial arrangement, speed and / or acceleration of the moving object 1400 adapt (e.g. with regard to up to three degrees of translation and up to three degrees of rotation).

The registration system 1408 can have one or more sensors that determine the spatial arrangement, speed and / or acceleration of the moving object 1400 can detect (e.g. in relation to up to three degrees of translation and up to three degrees of rotation). The one or more sensors can include global positioning system (GPS) sensors, motion sensors, inertial sensors, proximity sensors, or image sensors. The acquisition data received from the acquisition system 1408 can be used to determine the spatial arrangement, speed and / or orientation of the moving object 1400 to be controlled (e.g. using a suitable processing unit and / or a control module, as described below). Alternatively, the acquisition system 1408 can be used to provide data relating to the environment surrounding the moving object, such as weather conditions, proximity to potential obstacles, location of geographic features, location of artificial structures and the like.

The communication system 1410 enables communication with the end device 1412 that is a communication system 1414 has, via wireless signals 1416 . The communication systems 1410 , 1414 may have any number of transmitters, receivers and / or transceivers suitable for wireless communication. The communication can be one-way communication so that data can only be transmitted in one direction. For example, one-way communication can only be the moving object 1400 involve the data to the terminal 1412 sends, or vice versa. The data can be from one or more transmitters of the communication system 1410 to one or more recipients of the communication system 1412 sent, or vice versa. Alternatively, the communication can be two-way communication, allowing data to flow in both directions between the moving object 1400 and the end device 1412 can be sent. The two-way communication can involve data from one or more transmitters of the communication system 1410 to one or more recipients of the communication system 1414 to send and vice versa.

In some embodiments, the terminal can 1412 Control data to one or more of the moving object 1400 , Carrier 1402 and the payload 104 provide and information from one or more of the moving object 1400 , Carrier 1402 and the payload 1404 receive (e.g. position and / or movement information of the moving object, carrier or payload; data that is captured by the payload, such as image data that is recorded by a payload camera). In some examples, control data from the terminal device may include instructions for relative positions, movements, actuations or controls of the moving object, carrier and / or the payload. For example, the control data can be used in a modification of the location and / or the orientation of the moving object (e.g. via control of the drive mechanisms 1406 ) or a movement of the payload in relation to the moving object (e.g. via control of the carrier 1402 ) result. The control data from the terminal can result in control of the payload, such as controlling the operation of a camera or other image recording device (e.g. recording still or moving images, enlarging or reducing, switching on or off, switching imaging modes, changing image resolution, changing focus, Changing the depth of field, changing the exposure time, changing the viewing angle or field of view). In some examples, communications from the moving object, carrier, and / or payload may include information from one or more sensors (eg, from the sensing system 1408 or from the payload 1404 ). The communications may include captured information from one or more different types of sensors (e.g., GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors). Such information can relate to the position (e.g. location, orientation), movement or acceleration of the moving object, carrier and / or the payload. Such information from a payload can include data received by the payload or a detected state of the payload. The control data provided and received by the terminal 1412 can be configured to indicate a state of one or more of the moving objects, 1402 , Carrier 1402 or the payload 1404 to control. Alternatively or in combination, the carrier 1402 and the payload 1404 each also have a communication module that is configured with a terminal 1412 to communicate so that the terminal device with each of the moving objects 1400 , Carrier 1402 and the payload 1404 can communicate independently.

In some embodiments, the moving object can 1400 be configured with another remote device in addition to the terminal 1412 or instead of the terminal 1412 to communicate. The end device 1412 can also be configured with another remote device as well as the moving object 1400 to communicate. For example, the moving object can 1400 and / or terminal 1412 communicate with another moving object or a carrier or payload of another moving object. If desired, the remote device can be a second terminal or other computing device (e.g., computer, laptop, tablet, smartphone, or other mobile device). The remote device can be configured to send data to the moving object 1400 to send data from the moving object 1400 to receive data to the terminal 1412 to send and / or data from the terminal 1412 to recieve. Optionally, the remote device can be connected to the Internet or other telecommunications network so that data received from the moving object 1400 and / or terminal 1412 can be received, uploaded to a website or server.

22nd Figure 3 is a schematic illustration, by means of a block diagram, of a system 1500 for controlling a moving object in accordance with embodiments. The system 1500 can be used in combination with any suitable embodiment of the systems, devices, and methods disclosed herein. The system 1500 can be an acquisition module 1502 , a processing unit 1504 , a non-transitory computer readable medium 1506 , a control module 1508 and a communication module 1510 exhibit.

The acquisition module 1502 can use different types of sensors that collect information related to the moving objects in different ways. Different types of sensors can detect different types of signals or signals from different sources. For example, the sensors can have inertial sensors, GPS sensors, proximity sensors (eg lidar) or vision / image sensors (eg a camera). The acquisition module 1502 can be operational with a processing unit 1504 be coupled, which has a plurality of processors. In some embodiments, the acquisition module can be operable with a transmit module 1512 (e.g. a WiFi image transmission module) configured to send capture data directly to an appropriate external device or system. For example, the transmitter module 1512 used to capture images captured by a camera of the capture module 1502 be recorded to send to a remote terminal.

The processing unit 1504 may have one or more processors, such as a programmable processor (eg a central processing unit (CPU)). The processing unit 1504 may be operational on a non-transitory computer readable medium 1506 be coupled. The non-transitory computer-readable medium 1506 can store logic, code and / or program instructions issued by the processing unit 1504 are executable to perform one or more steps. The non-transitory computer readable medium may include one or more storage devices (e.g., removable media or external storage such as an SD card or random access memory (RAM)). In some embodiments, data can be obtained from the acquisition module 1502 directly to the storage units of the non-transitory computer-readable medium 1506 transferred and stored in it. The storage units of the non-transitory computer-readable medium 1506 can store logic, code and / or program instructions issued by the processing unit 1504 are operable to perform any suitable embodiment of the methods described herein. For example, the processing unit 1504 be configured to execute instructions that one or more processors of the processing unit 1504 cause to analyze acquisition data generated by the acquisition module. The storage units can store acquisition data from the acquisition module, which from the processing unit 1504 are to be processed. In some embodiments, the storage units of the non-transitory computer readable medium 1506 used to store the processing results obtained by the processing unit 1504 be generated.

In some embodiments, the processing unit 1504 operational with a control module 1508 be coupled configured to control a state of the moving object. For example, the control module 1508 be configured to control the drive mechanisms of the moving object in order to adjust the spatial arrangement, speed and / or acceleration of the moving object with respect to six degrees of freedom. Alternatively or in combination, the control module 1508 control one or more of a state of a carrier, payload, or acquisition module.

The processing unit 1504 can be operational with a communication module 1510 be coupled that is configured to send and / or receive data to and / or from one or more external devices (e.g., a terminal, display device, or other remote controller). Any suitable means of communication can be used, such as wired communication or wireless communication. For example, the communication module 1510 use one or more of local area networks (LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point (P2P) networks, telecommunication networks, cloud communication and the like. Optional relay stations such as masts, satellites or mobile stations can be used. Wireless communications can be proximity-dependent or proximity-independent. In some embodiments, line of sight may or may not be required for communication. The communication module 1510 can be one or more of acquisition data from the acquisition module 1502 , Processing results received from the processing unit 1504 are generated, send and / or receive predetermined control data, user commands from a terminal or a remote control, and the like.

The components of the system 1500 can be arranged in any suitable configuration. For example, one or more of the components of the system 1500 on the moving object, carrier, payload, terminal, detection system or an additional external device in communication with one or more of the previous ones. Additionally, though 22nd a single processing unit 1504 and a single non-transitory computer readable medium 1506 one skilled in the art will appreciate that this is not intended to be limiting and that the system 1500 may include a variety of processing units and / or non-transitory computer readable media. In some embodiments, one or more of the plurality of processing units and / or non-transitory computer-readable media may be in different locations, such as the moving object, the carrier, the payload, the terminal, the acquisition module, additional external devices in communication with one or more of the Previous, or suitable combinations thereof, so that any suitable aspect of the processing and / or storage functions performed by the system 1500 can be carried out at one or more of the aforementioned locations.

As used herein, A and / or B encompass one or more of A or B and combinations thereof, such as A and B.

The specific dimensions of any of the devices, devices, systems and components thereof of the present disclosure can already be varied depending on the envisaged application, as those skilled in the art will recognize in view of the disclosure herein. It is also understood that the embodiments described herein are for illustrative purposes only, and that various modifications or changes may be suggested to those skilled in the art and are encompassed within the spirit and scope of this application and the scope of the appended claims. Numerous different combinations of the embodiments described herein are possible and such combinations are considered part of the present disclosure. In addition, all of the features discussed in connection with any embodiment herein can be readily adapted for use in other embodiments herein.

While preferred embodiments of the present invention have been shown and described herein, those skilled in the art will recognize that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now become apparent to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein can be used in practicing the invention. It is contemplated that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered therein.

Claims (16)

A system for calibrating imaging devices, the system comprising: first and second imaging devices carried by an unmanned aerial vehicle, each imaging device configured to capture an image data set comprising a plurality of images (350, 352) of a calibration target (300, 500), the calibration target (300, 500) a plurality of features (302, 304) arranged in a repeating pattern and having a plurality of reference marks (502a, 502b, 502c), the reference marks (502a, 502b, 502c) in each image of the plurality of images (350, 352 ) are each uniquely identifiable, wherein the reference markings form vertices of an irregular polygon on the calibration target, wherein the one or more reference markings form vertices of an irregular polygon in the plurality of images and wherein the plurality of images (350, 352) comprises at least one image, that receives less than all of the plurality of features (302, 304); and one or more processors that are jointly or individually configured to: Receiving the plurality of images (350, 352) from each of the first and second imaging devices; Determining a spatial relationship between the reference marks (502a, 502b, 502c) in each image of the plurality of images (350, 352) and the reference marks (502a, 502b, 502c) on the calibration target (300, 500), wherein the step of determining the spatial relationship comprises identifying a first correspondence between the vertices of the irregular polygon in the plurality of images and the vertices of the irregular polygon on the calibration target, identifying corresponding reference marks based on the positioning of the reference marks relative to edges or Angling the polygon is performed; Formulating, based on the spatial relationship, a second correspondence between image coordinates of features in each image of the plurality of images (350, 352) and global coordinates for corresponding features on the calibration target (300, 500), based on the predetermined global coordinates of the reference marks (502a, 502b, 502c); and Determine calibration parameters for the first and second imaging device based on the second match. System according to Claim 1 wherein the system further comprises: processing one or more images obtained from each of the first and second imaging devices based on the calibration parameters to create a 3D representation of an environment of the unmanned aerial vehicle. System according to Claim 1 wherein each image of the plurality of images (350, 352) captures the reference marks (502a, 502b, 502c) and at least a subset of the plurality of features (302, 304) on the calibration target (300, 500). System according to Claim 1 wherein the plurality of images (350, 352) are captured by each of the first and second imaging devices at different positions and orientations relative to the calibration target (300, 500). System according to Claim 1 wherein the calibration target (300, 500) is a chessboard having a plurality of squares, and the plurality of features (302, 304) includes vertices of the plurality of squares. System according to Claim 5 wherein the plurality of features (302, 304) includes four outside corners of the chessboard and the at least one image omits at least one of the four outside corners. System according to Claim 1 wherein at least some of the angles of the irregular polygon have different sizes from other angles, or at least some of the edges of the irregular polygon have different lengths from other edges. System according to Claim 1 wherein the reference marks (502a, 502b, 502c) have three reference marks (502a, 502b, 502c) which form the vertices of an isosceles or unequal-sided triangle. System according to Claim 1 , wherein the one or more processors are further configured jointly or individually to: determine, with the aid of the one or more processors, correspondences between image coordinates of features via the image data sets, the position and / or orientation of the first and second imaging devices relative to one another appreciate. System according to Claim 1 wherein the plurality of features (302, 304) are arranged in the repeating pattern above a plane, and wherein the reference marks (502a, 502b, 502c) form a shape that is rotationally asymmetrical within the plane. System according to Claim 1 wherein the first imaging device and the second imaging device capture the images at substantially the same time. System according to Claim 1 wherein the formulating step comprises identifying features included in both the plurality of images (350, 352) obtained from the first camera and the plurality of images (350, 352) obtained from the second camera , available. System according to Claim 1 wherein the calibration parameters include at least one of a focal length, position of the image center point, pixel size, or optical distortion for each of the first and second imaging devices, or wherein the calibration parameters include a position and orientation of each of the first and second imaging devices relative to the calibration target. System according to Claim 1 , wherein each reference mark of the plurality of reference markings preferably has a different size, shape and / or color. System according to Claim 1 , wherein each reference mark of the plurality of reference marks has unique characteristics or identifications that enable the reference mark to be visually differentiated from the other reference marks of the plurality of reference marks, wherein preferably each reference mark of the plurality of reference marks has a different size, shape and / or color . Use of a system according to any preceding claim for calibrating imaging devices, using the system comprising: using the one or more processors of the system to apply an image dataset from each of first and second imaging devices carried by an unmanned aerial vehicle which has a plurality of images (350, 352) of a calibration target (300, 500), wherein the calibration target (300, 500) has a plurality of features (302, 304) arranged in a repeating pattern and a plurality of reference marks ( 502a, 502b, 502c), wherein the reference marks (502a, 502b, 502c) are each individually identifiable in each image of the plurality of images (350, 352), and wherein the reference marks form vertices of an irregular polygon on the calibration target, the one or more reference marks in vertices of an irregular polygon forming the plurality of images, the plurality of images (350, 352) including at least one image that captures less than all of the plurality of features (302, 304); Using the one or more processors of the system to establish a spatial relationship between the reference marks (502a, 502b, 502c) in each image of the plurality of images (350, 352) and the reference marks (502a, 502b, 502c) on the calibration target ( 300, 500), wherein the step of determining the spatial relationship comprises identifying a first correspondence between the vertices of the irregular polygon in the plurality of images and the vertices of the irregular polygon on the calibration target, identifying corresponding reference marks based on the Positioning of the reference marks relative to the edges or angles of the polygon is performed; Using the one or more processors of the system to determine, based on the spatial relationship, a second correspondence between image coordinates of features in each image of the plurality of Formulate images (350, 352) and global coordinates for corresponding features on the calibration target (300, 500); and using the one or more processors of the system to determine calibration parameters for the first and second imaging devices based on the second match.

Images