WO2011158344A1

WO2011158344A1 - Image processing method, program, image processing device, and imaging device

Info

Publication number: WO2011158344A1
Application number: PCT/JP2010/060185
Authority: WO
Inventors: 上田　滋之; 央樹坪井
Original assignee: コニカミノルタオプト株式会社
Priority date: 2010-06-16
Filing date: 2010-06-16
Publication date: 2011-12-22
Also published as: JPWO2011158344A1

Abstract

Disclosed is an image processing method for applying a specific distortion correction in less time with a comparatively small circuit. In said image processing method, a virtual projection surface set in a world coordinate system, said virtual projection surface comprising a curved surface or a plurality of flat surfaces joined together, is converted to a camera coordinate system by means of distortion-correction coefficients, and data for an image projected onto said virtual projection surface is computed on the basis of the converted camera-coordinate-system coordinates and pixel data from imaging elements.

Description

Image processing method, program, image processing apparatus, and imaging apparatus

The present invention relates to an image processing method, a program, an image processing apparatus, and an imaging apparatus that perform distortion correction processing on an image captured by an imaging element via an optical system including a condenser lens.

Generally, since an image taken by an optical system having a short focal length lens or a large angle of view lens such as a wide-angle lens or a fish-eye lens is distorted, image processing for correcting the distortion is performed. Patent Document 1 discloses a correction method of the prior art that corrects distortion generated in a captured image captured using a lens with a short focal length using a lens correction parameter.

In the display device that displays the periphery of a vehicle disclosed in Patent Document 2, the image height change rate increases when the incident angle is greater than or equal to a predetermined value in image processing when displaying photographing data photographed using a wide-angle lens on the display. In the case of less than a predetermined value, correction is performed so that the change rate of the image height decreases.

JP 2009-140066 A JP 2010-3014 A

As disclosed in

Patent Documents

1 and 2, when the captured image obtained by the lens is linearly corrected, the image is enlarged in the peripheral portion as compared with the central portion, so that the image is deformed and is difficult to see. Corrections to prevent this include cylindrical distortion correction and spherical distortion correction. However, when hardware is used as an image processing apparatus for performing these corrections, the processing time becomes long, the circuit scale increases, and the cost increases. There was a problem that increased.

In particular, in a case where a captured image is displayed on a monitor in real time, such as a surveillance camera or an in-vehicle camera disclosed in Patent Document 2, zoom, pan, tilt, etc. are performed based on an instruction to change a shooting area by a user. Each time processing is added, new image processing is required.

For this reason, there is a problem that when the image processing apparatus is implemented as hardware, the processing time becomes long, the circuit scale increases, and the cost increases.

In view of such a problem, the present invention provides an image processing method, a program, an image processing apparatus, and an imaging apparatus capable of performing specific distortion processing while reducing processing time with a relatively small circuit. The purpose is to provide.

The above object is achieved by the invention described below.

1. In an image processing method for obtaining image data that has been subjected to distortion correction processing using a plurality of pixel data obtained by receiving light on an image sensor having a plurality of pixels via an optical system including a lens,
A first step of setting a surface shape, a size, and a position in a world coordinate system of a virtual projection surface composed of a curved surface or a surface obtained by connecting a plurality of planes based on a user instruction;
A second step of converting coordinates in the world coordinate system of each pixel of the virtual projection plane set in the first step into a camera coordinate system using a distortion correction coefficient of the optical system;
A third step of calculating image data of the virtual projection plane set in the first step based on the plurality of pixel data and the coordinates in the camera coordinate system converted in the second step;
An image processing method comprising:

2. 2. The image processing according to 1, wherein the surface shape of the virtual projection plane set in the first step is a cylindrical shape having a circular cross section or a polygonal mirror shape including a plurality of continuous panels. Method.

3. 3. The image processing method according to 1 or 2, further comprising a fourth step of displaying the image data calculated in the third step on a display unit.

4. The image processing method according to any one of claims 1 to 3, wherein the virtual projection plane set in the first step is a plurality of virtual projection planes.

5. In the third step, the image data is calculated from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane, and the corresponding position on the imaging element surface is calculated. 5. The image processing method according to claim 1, wherein image data at each position is obtained from pixel data.

6). An image processing apparatus for obtaining image data that has been subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A virtual projection plane in the world coordinate system, which converts the coordinates in the world coordinate system of each pixel of the virtual projection plane consisting of a curved surface or a plane connecting a plurality of planes to the camera coordinate system using the distortion correction coefficient of the optical system An image processing unit that calculates image data of the virtual projection plane set by the setting unit based on the coordinates converted into the camera coordinate system and the plurality of pixel data;
An image processing apparatus comprising:

7. 7. The image processing apparatus according to 6, wherein the shape of the virtual projection surface is a cylindrical shape having a circular arc cross section or a polygonal mirror shape including a plurality of continuous panels.

8. 8. The image processing apparatus according to 6 or 7, wherein the virtual projection plane is a plurality of virtual projection planes.

9. The image processing unit calculates the position of the corresponding pixel on the image sensor surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. The image processing apparatus according to any one of 6 to 8, wherein image data at each position is obtained from pixel data.

10. An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
A virtual projection plane of a world coordinate system in which a position and a size are set, and the coordinates in the world coordinate system of each pixel of the virtual projection plane formed by a curved surface or a plane connecting a plurality of planes is used as a distortion correction coefficient of the optical system. An image processing unit for calculating image data on the virtual projection plane based on the coordinates converted into the camera coordinate system and the plurality of pixel data.
Program to function as.

11. An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
A setting unit capable of setting a shape, a size, and an arrangement position of a virtual projection plane, which is a virtual projection plane in the world coordinate system, and includes a curved surface or a plane connecting a plurality of planes;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data Based on the image processing unit for calculating the image data of the virtual projection plane set by the setting unit,
Program to function as.

12 Optical system,
An imaging device having a plurality of pixels;
A setting unit capable of setting a shape, a size, and an arrangement position of a virtual projection plane, which is a virtual projection plane of a world coordinate system, which is a curved surface or a plane connecting a plurality of planes;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, and the coordinates converted into the camera coordinate system and the image sensor are received. An image processing unit that calculates image data of a virtual projection plane set by the setting unit based on a plurality of pixel data obtained in the above manner;
An imaging device comprising:

13. 13. The imaging apparatus according to 12, wherein the shape of the virtual projection surface is a cylindrical shape having a circular arc cross section or a polygonal mirror shape including a plurality of continuous panels.

14 An operation unit operated by a user;
A display unit;
Have
The setting unit sets the shape, size, and position of the virtual projection plane in the world coordinate system based on an operation to the operation unit,
The image pickup apparatus according to item 12 or 13, wherein the display unit displays image data on the set virtual projection plane.

15. 15. The imaging apparatus according to any one of 12 to 14, wherein the virtual projection plane set by the setting unit is a plurality of virtual projection planes.

16. The image processing unit calculates the position of the corresponding pixel on the image sensor surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. The image pickup apparatus according to any one of 12 to 15, wherein image data at each position is obtained from pixel data.

According to the present invention, the virtual projection plane set in the world coordinate system is converted into the camera coordinate system using the distortion correction coefficient, and the virtual projection plane is based on the converted coordinates of the camera coordinate system and the pixel data of the image sensor. By calculating the image data, it is possible to reduce the processing time with a relatively small circuit.

In particular, the virtual projection plane is composed of a curved surface or a plane connecting a plurality of planes, so that cylindrical distortion correction, spherical distortion correction, and trihedral mirror distortion correction can be handled in a single process.

It is a schematic diagram explaining the distortion correction which concerns on this embodiment. An example in which the position of the virtual projection plane VP is moved is shown. It is a block diagram which shows schematic structure of an imaging device. It is a figure which shows the main control flow. It is a figure which shows the subroutine of step S10. It is a figure which shows the subroutine of step S20. FIG. 7A is a diagram illustrating an example of the shape of a virtual projection plane, FIG. 7A is an example of a cylindrical shape, and FIG. 7B is an example of a trihedral mirror shape. It is a figure explaining the coordinate of the virtual projection surface VP. It is a figure which shows the correspondence of camera coordinate system xy and image pick-up element surface IA. An example in which two virtual projection planes VP are set is shown. This is an example in which the position of the virtual projection plane VP is changed with the image center o of the camera coordinate system as the rotation center. An example in which the virtual projection plane VP0 is rotated by roll is shown. An example in which the virtual projection plane VP0 is pitch rotated is shown. An example in which the virtual projection plane VP0 is rotated by yaw is shown. This is an example in which the position of the virtual projection plane VP is changed with the center ov of the virtual projection plane VP0 as the rotation center. An example in which the virtual projection plane VP0 is rotated by a virtual pitch is shown. An example in which the virtual projection plane VP0 is rotated by a virtual yaw is shown. This is an example in which the position of the virtual projection plane VP is changed with the image center o in the camera coordinate system as the movement center. An example in which the virtual projection plane VP0 is offset in the X direction is shown. An example in which the virtual projection plane VP0 is offset in the Y direction is shown. An example in which the virtual projection plane VP0 is offset in the Z direction is shown. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. An example in which the shape of the virtual projection plane VP is a cylindrical shape in the reverse direction is shown. FIG. 25 is an example in which image data calculated using a virtual projection plane corresponding to FIG. 24 is displayed on the display unit 120. FIG.

The present invention will be described based on an embodiment, but the present invention is not limited to the embodiment.

FIG. 1 is a schematic diagram for explaining distortion correction according to the present embodiment. In FIG. 1, X, Y, and Z are world coordinate systems, and the origin O is the lens center. Z includes the optical axis, and the XY plane includes the lens center plane LC passing through the lens center O. Point P is an object point of the object in the world coordinate system XYZ. θ is an incident angle with respect to the optical axis (coincident with the Z axis).

X and y are camera coordinate systems, and the xy plane corresponds to the image sensor surface IA. o is the center of the image and is the intersection of the optical axis Z and the image sensor surface. The point p is a point on the image sensor surface in the camera coordinate system, and the object point P is converted into the camera coordinate system using a distortion correction coefficient based on a parameter based on lens characteristics (hereinafter referred to as “lens parameter”). is there.

VP is a virtual projection plane, and consists of a curved surface or a surface connecting a plurality of planes. The virtual projection plane VP is set on the opposite side of the imaging element (and imaging element surface IA) with respect to the lens position (lens center plane LC) of the optical system. The virtual projection plane VP can be changed in shape, size, and position in the world coordinate system based on an instruction from the user to the operation unit 130 (see FIG. 3).

In the present application, “position change” is a concept that includes not only the case where the virtual projection plane VP is translated on the XY plane, but also an angle change (also referred to as an attitude change) with respect to the XY plane.

“Cylinder distortion correction (or cylinder correction)” is a distortion correction method performed using the following virtual projection plane VP. The virtual projection plane VP shape is rectangular or square for the shape projected on the XY plane perpendicular to the optical axis Z direction (the X axis is the horizontal direction and the Y axis is the vertical direction), and is not constant in the optical axis Z direction, There are arbitrary changes.

“Spherical distortion correction (or spherical correction)” is a distortion correction method performed using the following virtual projection plane VP. The virtual projection plane VP shape is a shape in which the inner surface of the sphere is cut out into a rectangle, and the shape projected on the XY plane perpendicular to the optical axis Z is a rectangle or square, and the X axis direction and the Y axis direction are also Z. The axial direction is not constant and is arbitrarily changed. In particular, the coordinate unit change amount in the X-axis direction and the Y-axis direction has a constant curvature.

“Trihedral mirror correction (or trihedral mirror distortion correction)” is a distortion correction method performed using the following virtual projection plane VP. The virtual projection plane VP shape is arbitrary in the X-axis direction and Y-axis direction (particularly preferably rectangular or square), a part is constant (no change) in the optical axis Z direction, There are arbitrary changes.

In the example shown in FIG. 1, an example of a cylindrical shape in which a part of the inner peripheral surface of the cylinder is cut off as the virtual projection surface is shown. The shape of the virtual projection plane is not limited to this, and may be a shape of the inner peripheral surface of a hemisphere or a spherical crown cut out by a plane intersecting with the sphere, or a shape obtained by cutting the inner peripheral surface of the sphere into a rectangle. . Moreover, the shape which connected the some surface may be sufficient. As a shape in which a plurality of surfaces are connected, a three-sided mirror shape (described later) composed of three continuous panels is preferable.

In the initial state (initial position setting, the same applies hereinafter), the virtual projection plane VP has a predetermined shape and size, and the center ov of the virtual projection plane VP is located on the Z axis. Gv is a point where the object point P is projected onto the virtual projection plane VP, and is an intersection of the object point P and a straight line passing through the lens center O and the virtual projection plane VP. A virtual projection plane VP1 in FIG. 2 shows a state in which the virtual projection plane VP0 is rotated on the XZ plane based on the input of the operation unit 130.

[Block Diagram]
FIG. 3 is a block diagram illustrating a schematic configuration of the imaging apparatus. The imaging apparatus includes an imaging unit 110, a control device 100, a display unit 120, and an operation unit 130.

The imaging unit 110 includes a short-focus lens, an imaging element, and the like. In the present embodiment, in the present embodiment, examples of the lens include a wide-angle lens and a fisheye lens.

The control device 100 includes an image processing unit 101, a setting unit 102, and a storage unit 103.

The setting unit 102 sets the shape, position, and size of the virtual projection plane VP based on an input instruction to the operation unit 130.

The image processing unit 101 creates a conversion table of each coordinate on the virtual projection plane into the camera coordinate system based on the set shape, position, and size of the virtual projection plane VP, and uses the conversion table to capture the imaging unit 110. The pixel data photographed in the above is processed to generate image data to be displayed on the display unit 120. The storage unit 103 stores a distortion correction coefficient calculated based on the lens parameters of the lens. Also, the position and size of the virtual projection plane VP and the created conversion table are stored.

The display unit 120 includes a display screen such as a liquid crystal display, and sequentially displays the image data created by the image processing unit 101 based on the pixel data captured by the imaging unit 110 on the display screen.

The operation unit 130 includes a keyboard, a mouse, or a touch panel arranged so as to be superimposed on the liquid crystal display of the display unit, and receives a user's input operation.

[Control flow]
4 to 6 are diagrams showing a control flow of the present embodiment. FIG. 4 is a diagram showing a main control flow, and FIGS. 5 and 6 are diagrams showing a subroutine of FIG.

In step S10 (corresponding to “first step”), the virtual projection plane VP is converted (set). The virtual projection plane conversion in step S10 will be described with reference to FIG. In step S11, the type of virtual projection plane shape is selected by the user. As the types, there are a cylindrical shape and a three-sided mirror shape as described above.

In step S12, the shape and position of the virtual projection plane are set. FIG. 7 is a diagram showing an example of the shape of the virtual projection plane, FIG. 7A is an example in which a cylindrical shape is selected as the shape of the virtual projection plane, and FIG. 7B is a diagram in which a trihedral mirror shape is selected. This is an example. In the former, cylindrical distortion correction is performed, and in the latter, trihedral mirror distortion correction is performed.

As shown in FIGS. 7A and 7B, in the initial state, the longitudinal direction is parallel to the X axis of the world coordinate system, and the short side direction is parallel to the Y axis. The surface near the end is inclined inward. The center ov of the virtual projection plane VP is located on the Z axis, and the center axis v-axis passing through the center ov is on the XY plane. Further, both virtual projection planes VP have a symmetrical shape about the central axis ov-axis. By making it symmetrical, the memory capacity required to store the virtual projection plane VP can be reduced. Note that the inclination of the end portion is not limited to the inside (a shape convex toward the object point side), and may be a shape inclined toward the outside (a shape convex toward the image side) (see FIG. 24).

The virtual projection plane VP shown in FIG. 7 (a) has a cylindrical shape with both end portions inclined inward, and has a shape in which the inward inclination angle increases as the distance from the central axis v-axis increases. In the figure, the length l _X, l _Y, set the shape of the virtual projection plane VP by setting the l _Z is performed, it is possible to set the position by changing the position of the ov. In the initial state, the sides corresponding to the lengths l _X , l _Y , and l _Z are parallel to the X axis, the Y axis, and the Z axis, respectively. The camera viewpoint displayed on the display unit 120 is changed by changing the position as shown in FIG. 2 (corresponding to pan and tilt). Further, if the distance from the lens center O is changed with the change of the position of the virtual projection plane VP, the zoom-in / zoom-out is performed. The vertical and horizontal lengths l _X, since the angle of view is changed by changing the l _Y can be obtained zoom, the same effects and zoom-out. Since the curvature of the virtual projection plane VP is changed by changing the depth direction of the length l _Z, the strength of the cylindrical distortion correction due to this is changed. A specific example regarding the position change of the virtual projection plane VP will be described later.

The virtual projection plane VP shown in FIG. 7B has a three-sided mirror shape in which both ends are inclined inward, and three panels of the main mirror part m and the side mirror parts S1 and S2 are continuous. In the initial state, the plane of the main mirror part m is parallel to the XY plane, and the sides corresponding to the lengths l _X1 and l _X2 , l _Y1 and l _Z1 are parallel to the X axis, Y axis and Z axis, respectively. is there. The shape of the side mirror parts S1 and S2 and the angle of the plane with respect to the plane of the main mirror part m are the same, and the virtual projection plane VP has a bilaterally symmetric shape about the central axis v-axis. By setting the lengths l _X1 , l _X2 , l _Y1 , and l _Z1 , the shape of the virtual projection plane VP is set, and the position can be set by changing the position of ov. In the figure, an example of a three-sided mirror in which two side mirror parts are connected to both sides of the main mirror part is shown, but not only on both sides (X axis direction) of the main mirror part but also in the vertical direction (Y axis). The shape may be a polygonal mirror in which side mirrors are connected in the direction).

Returning to the explanation of FIG. In step S13, the virtual projection plane VP set up to step S12 is divided into n pixels. The number of pixels is preferably equal to or greater than the total number of display pixels (screen resolution) of the display unit 120. Hereinafter, as an example, both the number of pixels n and the total number of display pixels of the display unit 120 will be described under a fixed condition of 640 × 480 pixels (total number of pixels 307,000).

In the present embodiment, the interval between adjacent pixels on the virtual projection plane VP is set to be equal. Further, the virtual projection plane VP has a shape formed by a curved surface or a surface obtained by connecting a plurality of planes, and a shape in which the end side surface is inclined. For these reasons, the virtual projection plane VP is inclined as compared with the periphery of the center ov that is more directly opposed to the lens center plane LC (the main mirror portion m in the example of FIG. 7B). Pixels are arranged at higher density on both end sides (in the example of FIG. 7B, the side mirror portions S1 and S2) with respect to the change in the incident angle θ. As a result, it is possible to obtain an easily recognizable image with little influence on the shape and color with respect to stretching when performing distortion correction. This is the control related to the subroutine of step S10 shown in FIG.

In step S20 of FIG. 4 (corresponding to “second and third steps”), distortion correction processing is mainly performed by the image processing unit 101 based on the virtual projection plane VP set in step S10. The distortion correction process in step S20 will be described with reference to FIG.

In step S21, the coordinates Gv (X, Y, Z) of the world coordinate system are acquired for each pixel Gv on the virtual projection plane VP. FIG. 8 is a schematic diagram for explaining a coordinate system. As shown in FIG. 8, point A (0, 0, Za), point B (0, 479, Zb), point C (639, 479, Zc), point D (639, 0) at the four corners of the virtual projection plane VP. , Zd) is divided into 640 × 480 pixel pixels Gv (total number of pixels: 307,000) at equal intervals, and the coordinates of all the pixels Gv in the world coordinate system are obtained.

When acquiring the world coordinate system, if the virtual projection surface VP has a curved surface such as a cylindrical shape, the inclination of each panel is determined based on the curvature of the virtual projection surface VP if a plurality of panels such as a three-surface shape are continuous. Calculate based on the angle.

In step S22, coordinates Gi (x, y in the corresponding camera coordinate system on the image sensor surface IA are calculated from the coordinates of the pixel Gv in the world coordinate system and the distortion correction coefficient of the imaging unit 110 stored in the storage unit 103. ) Is calculated. Specifically, the distortion correction coefficient calculated from the lens parameters of the optical system is stored in the storage unit 103, and is calculated from the incident angle θ with respect to the optical axis Z obtained from the coefficient and the coordinates of each pixel Gv. (Reference: International Publication No. 2010/032720).

FIG. 9 is a diagram showing a correspondence relationship between the camera coordinate system xy and the imaging element surface IA. In FIG. 9, points a to d are obtained by converting the points A to D in FIG. 8 into the camera coordinate system. In FIG. 8, the virtual projection plane VP surrounded by the points A to D is a rectangular plane, but in FIG. 9, the area surrounded by the points a to d after the coordinate transformation to the camera coordinate system is (the virtual projection plane VP It becomes a distorted shape (corresponding to the position and shape). The figure shows an example of distortion in a barrel shape. However, depending on the characteristics of the optical system, it may be a pincushion type or a Jinkasa type (a shape that changes into a barrel shape at the center and straight or pincushion at the end). There is also.

In step S23, the pixel of the imaging device to be referred to is determined from the coordinates Gi (x ', y'). Note that x and y in the coordinates (x, y) of each pixel of the image sensor are integers, but x ′ and y ′ in the coordinates Gi (x ′, y ′) calculated in step S22 are not necessarily integers. Can take a real value with a fractional part. When x ′ and y ′ are integers as in the former and the coordinates Gi (x ′, y ′) coincide with the pixel position of the image sensor, the pixel data of the pixel of the corresponding image sensor is used as the virtual projection plane. Used as pixel data of the pixel Gv (X, Y, Z) on the VP. As in the latter case, when x ′ and y ′ are not integers and x ′ and y ′ do not match x and y, the pixel data of the pixel Gv is calculated as the pixel data around the calculated coordinate Gi (x ′, y ′). Using the pixel data of the pixels, for example, the top four pixels close to the position of the coordinate Gi (x ′, y ′), they are closer to these simple average values or the distance to the coordinate Gi (x ′, y ′). Pixel data calculated by weighting four pixels may be used. The peripheral locations are not limited to 4 locations, and may be 1 location, 16 locations or more.

Steps S21 to S24 are moved from the point A (0, 0, Za), which is the starting point of FIG. 8, by one pixel (pixel) at a time until each pixel ( (Pixel), image data in which distortion correction has been performed for all pixels can be acquired. This is the control related to the subroutine of step S20 shown in FIG.

Returning to the explanation of the control flow in FIG. In step S40 (corresponding to “fourth step”), the image data acquired in step S20 is displayed on the display screen of the display unit 120. Steps S20 and S40 are sequentially performed, and the image data after distortion processing based on the captured pixel data is displayed on the display unit 120 in real time.

According to the present embodiment, by setting the shape, position, and size of the virtual projection plane VP, it is possible to handle all processes including panning, tilting, and zooming processes including distortion correction processes in a batch process. Therefore, the processing becomes light and the processing time can be shortened with a relatively small circuit. In particular, by making the virtual projection plane VP into a shape formed by a curved surface or a surface obtained by connecting a plurality of planes, cylindrical distortion correction, spherical distortion correction, and trihedral mirror distortion correction can be handled in a single process. More specifically, conventionally, image data is calculated, panned, tilted, zoomed processing, cylindrical distortion correction, and processed image data is obtained through a series of multiple stages of processing, but in this embodiment, All of these series of processes can be performed as a batch process. Thus, even a relatively small circuit configuration such as an ASIC (Application Specific Integrated Circuit) can be processed in real time.
[Second Embodiment]
FIG. 10 is a schematic diagram illustrating distortion correction according to the second embodiment. In the description of FIGS. 1 to 6, the single virtual projection plane is used. However, the virtual projection plane is not limited to this and may be two or more. The image data obtained at the positions of the respective virtual projection planes are switched and displayed by time division or a user instruction, or the display screen is divided and displayed at the same time. The second embodiment shown in FIG. 10 is an example in which two virtual projection planes are set. Except for the configuration shown in FIG. 10, it is the same as the embodiment described in FIGS.

FIG. 10 shows an example in which two virtual projection planes VPh and VPj are set. Both can set the shape, position, and size independently. In the drawing, the ranges corresponding to the virtual projection planes VPh and VPj on the image sensor surface IA are the areas h and j, and the points corresponding to the object points P1 and P2 are the points p1 and p2.

Image data is calculated for each of the set virtual projection planes VPh and VPj by the flow shown in FIGS. 4 to 6, and each is individually displayed on the display unit 120 by the process of step S40 of FIG. The

[Specific Example of Changing Position of Virtual Projection Plane VP]
11 to 14 are examples in which the position of the virtual projection plane VP is changed with the image center o in the camera coordinate system as the rotation center (or the movement center). As shown in FIG. 11, rotation about the x-axis with the image center o as the rotation center is pitch (also referred to as tilt), rotation about the y-axis is yaw (also referred to as pan), and rotation about the Z-axis. Is roll.

FIGS. 12, 13, and 14 show examples in which the virtual projection plane VP0 is rotated by roll, pitch, and yaw, respectively, based on the input rotation amount setting value. In these figures (and also thereafter), Ca0 and Ca1 are virtual cameras, and cav is the camera viewpoint of the virtual camera Ca1 after the position change. Note that the camera viewpoint cav of the virtual camera Ca0 before the position change coincides with the Z axis.

In FIG. 12, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating it. The camera viewpoint cav coincides with before and after the position change.

In FIG. 13, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating the pitch.

14, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating the yaw, and the camera viewpoint cav is rotating clockwise in FIG.

15 to 17 are examples in which the position of the virtual projection plane VP is changed based on the center ov of the virtual projection plane VP0 based on the input rotation amount setting value. In the following, viewpoint conversion corresponding to rotating or changing the position of the virtual camera Ca0 is performed.

As shown in FIG. 15, one of two axes that are orthogonal to each other on the virtual projection plane VP0 is set as Yaw-axis, and the other is set as P-axis. Both are axes passing through the center ov, and rotation around the Yaw-axis with the center ov as the rotation center is called virtual yaw rotation, and rotation around the P-axis is called virtual pitch rotation. In the initial state, the virtual projection plane VP0 is parallel to the xy plane of the camera coordinate system, and the center ov exists on the Z axis. In this initial state, P-axis is parallel to the x axis, and Yaw-axis is parallel to the y axis (note that Yaw-axis and v-axis in FIG. 7 are the same).

In FIG. 16, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating the virtual pitch, and after the position change, the virtual camera Ca1 is positioned above, and the camera viewpoint cav is in a direction to look down.

17, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating the virtual yaw, and the virtual camera Ca1 and the camera viewpoint cav rotate counterclockwise after the position change.

FIG. 18 is an example in which the position of the virtual projection plane VP0 is changed with the image center o in the camera coordinate system as the movement center. In these examples, viewpoint conversion corresponding to parallel translation of the virtual camera Ca0 together with the virtual projection plane VP0 is performed.

19, 20, and 21 show examples in which the virtual projection plane VP0 is offset-shifted (translated) in the X, Y, and Z directions based on the input offset movement amount setting value. is there. In the initial state, the offset movement in the Z direction is the same movement as zooming in and zooming out. Except in the initial state, the offset movement in each direction is effective when moving the dark part outside the imaging area of the optical system outside the image area.

22 and 23 are examples of display images displayed on the display unit 120. FIG. FIG. 22 is an example of a display image that displays image data calculated based on a virtual projection plane VP of a cylindrical shape (FIG. 1, FIG. 7A, etc.). FIG. It is an example of the display image in the virtual projection surface VP of b)). An image close to the center can be accurately recognized with little distortion, and an image close to the left and right ends is enlarged and displayed. Since cylindrical distortion correction or trihedral mirror distortion correction is performed, the shape is unlikely to be deformed as the image is enlarged on the end side.

FIG. 24 shows a modification of the shape of the virtual projection plane VP. In the example shown in the figure, the virtual projection plane VP has a reverse cylindrical shape, and both end sides are inclined outward (back side).

FIG. 25 is an example of a display image that displays image data calculated based on the virtual projection plane VP having the reverse cylindrical shape shown in FIG. Although an image close to the center has little distortion, an image close to the left and right ends can display a wide area although distortion remains. Even in the virtual projection plane having such a shape, the pixels are arranged at a high density with respect to the change in the incident angle θ on the side closer to the left and right ends than the center side. The phenomenon that the image becomes rough on the side can be avoided.

In the embodiment described above, the optical system including a single lens is exemplified as the optical system including the condensing lens. However, the condensing lens may be composed of a plurality of lenses. Of course, an optical element other than the condenser lens may be provided, and the invention of the present application can be applied. In that case, the distortion correction coefficient may be a value for the entire optical system. However, in terms of downsizing and cost reduction of the imaging device, it is most preferable to use a single lens as exemplified in the above embodiment.

DESCRIPTION OF SYMBOLS 100 Control apparatus 101 Image processing part 102 Setting part 103 Storage part 110 Imaging unit 120 Display part 130 Operation part VP Virtual projection plane LC Lens center plane IA Image sensor surface O Lens center o Image center ov Center of virtual projection plane

Claims

In an image processing method for obtaining image data that has been subjected to distortion correction processing using a plurality of pixel data obtained by receiving light on an image sensor having a plurality of pixels via an optical system including a lens,
A first step of setting a surface shape, a size, and a position in a world coordinate system of a virtual projection surface composed of a curved surface or a surface obtained by connecting a plurality of planes based on a user instruction;
A second step of converting coordinates in the world coordinate system of each pixel of the virtual projection plane set in the first step into a camera coordinate system using a distortion correction coefficient of the optical system;
A third step of calculating image data of the virtual projection plane set in the first step based on the plurality of pixel data and the coordinates in the camera coordinate system converted in the second step;
An image processing method comprising:
2. The image according to claim 1, wherein the surface shape of the virtual projection plane set in the first step is a cylindrical shape having a circular cross section or a polygonal mirror shape including a plurality of continuous panels. Processing method.
3. The image processing method according to claim 1, further comprising a fourth step of displaying the image data calculated in the third step on a display unit.
The image processing method according to any one of claims 1 to 3, wherein the virtual projection plane set in the first step is a plurality of virtual projection planes.
In the third step, the image data is calculated from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane, and the corresponding position on the imaging element surface is calculated. The image processing method according to claim 1, wherein image data at each position is obtained from pixel data.
An image processing apparatus for obtaining image data that has been subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A virtual projection plane in the world coordinate system, which converts the coordinates in the world coordinate system of each pixel of the virtual projection plane consisting of a curved surface or a plane connecting a plurality of planes to the camera coordinate system using the distortion correction coefficient of the optical system An image processing unit that calculates image data of the virtual projection plane set by the setting unit based on the coordinates converted into the camera coordinate system and the plurality of pixel data;
An image processing apparatus comprising:
The image processing apparatus according to claim 6, wherein the shape of the virtual projection surface is a cylindrical shape having a circular cross section or a polygonal mirror shape including a plurality of continuous panels.
The image processing apparatus according to claim 6 or 7, wherein the virtual projection plane is a plurality of virtual projection planes.
The image processing unit calculates the position of the corresponding pixel on the image sensor surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. 9. The image processing apparatus according to claim 6, wherein image data at each position is obtained from pixel data.
An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
A virtual projection plane of a world coordinate system in which a position and a size are set, and the coordinates in the world coordinate system of each pixel of the virtual projection plane formed by a curved surface or a plane connecting a plurality of planes is used as a distortion correction coefficient of the optical system. An image processing unit for calculating image data on the virtual projection plane based on the coordinates converted into the camera coordinate system and the plurality of pixel data.
Program to function as.
An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
A setting unit capable of setting a shape, a size, and an arrangement position of a virtual projection plane, which is a virtual projection plane in the world coordinate system, and includes a curved surface or a plane connecting a plurality of planes;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data Based on the image processing unit for calculating the image data of the virtual projection plane set by the setting unit,
Program to function as.
Optical system,
An imaging device having a plurality of pixels;
A setting unit capable of setting a shape, a size, and an arrangement position of a virtual projection plane, which is a virtual projection plane of a world coordinate system, which is a curved surface or a plane connecting a plurality of planes;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, and the coordinates converted into the camera coordinate system and the image sensor are received. An image processing unit that calculates image data of a virtual projection plane set by the setting unit based on a plurality of pixel data obtained in the above manner;
An imaging device comprising:
13. The imaging apparatus according to claim 12, wherein the shape of the virtual projection plane is a cylindrical shape having a circular cross section or a polygonal mirror shape including a plurality of continuous panels.
An operation unit operated by a user;
A display unit;
Have
The setting unit sets the shape, size, and position of the virtual projection plane in the world coordinate system based on an operation to the operation unit,
The imaging device according to claim 12 or 13, wherein the display unit displays image data on the set virtual projection plane.
15. The imaging apparatus according to claim 12, wherein the virtual projection plane set by the setting unit is a plurality of virtual projection planes.
The image processing unit calculates the position of the corresponding pixel on the image sensor surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. The image pickup apparatus according to claim 12, wherein image data at each position is obtained from pixel data.