WO2011158343A1

WO2011158343A1 - Image processing method, program, image processing device, and imaging device

Info

Publication number: WO2011158343A1
Application number: PCT/JP2010/060184
Authority: WO
Inventors: 上田　滋之; 央樹坪井
Original assignee: コニカミノルタオプト株式会社
Priority date: 2010-06-16
Filing date: 2010-06-16
Publication date: 2011-12-22
Also published as: JPWO2011158343A1

Abstract

In order to reduce distortion correction processing time in a relatively small-scale circuit, the disclosed image processing method converts a virtual projection surface to a camera coordinate system using distortion correction coefficients, and calculates image data of the virtual projection surface set in a world coordinate system on the basis of the converted camera coordinate system coordinates and the pixel data of an imaging element.

Description

Image processing method, program, image processing apparatus, and imaging apparatus

The present invention relates to an image processing method, a program, an image processing apparatus, and an imaging apparatus that perform distortion correction processing on an image captured by an imaging element via an optical system including a condenser lens.

Generally, since an image taken by an optical system having a short focal length lens or a large angle of view lens such as a wide-angle lens or a fish-eye lens is distorted, image processing for correcting the distortion is performed. Patent Document 1 discloses a correction method of the prior art that corrects distortion generated in a captured image captured using a lens with a short focal length using a lens correction parameter.

In the display device that displays the periphery of a vehicle disclosed in Patent Document 2, the image height change rate increases when the incident angle is greater than or equal to a predetermined value in image processing when displaying photographing data photographed using a wide-angle lens on the display. In the case of less than a predetermined value, correction is performed so that the change rate of the image height decreases.

JP 2009-140066 A JP 2010-3014 A

Image processing for captured images obtained with lenses as disclosed in

Patent Documents

1 and 2 requires a lot of correction processing such as shading correction and distortion correction, and is therefore processed when hardware is used as an image processing apparatus. There is a problem that the time is increased, the circuit scale is increased, and the cost is increased.

In particular, when a captured image is displayed on a monitor in real time like a surveillance camera or an in-vehicle camera disclosed in Patent Document 2, zooming, panning, tilting, and the like are performed based on an instruction to change a shooting area by a user. Each time processing is added, new image processing is required.

For this reason, there is a problem that when the image processing apparatus is implemented as hardware, the processing time becomes long, the circuit scale increases, and the cost increases. SUMMARY OF THE INVENTION In view of such problems, the present invention has an object to provide an image processing method, a program, an image processing apparatus, and an imaging apparatus capable of reducing processing time with a relatively small circuit. .

The above object is achieved by the invention described below.

1. In an image processing method for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A first step of setting the position and size of the virtual projection plane in the world coordinate system based on a user instruction;
A second step of converting coordinates in the world coordinate system of each pixel of the virtual projection plane set in the first step into a camera coordinate system using a distortion correction coefficient of the optical system;
A third step of calculating image data of the virtual projection plane set in the first step based on the plurality of pixel data and the coordinates in the camera coordinate system converted in the second step;
An image processing method comprising:

2. 2. The image processing method according to item 1, further comprising a fourth step of displaying the image data calculated in the third step on a display unit.

3. 3. The image processing method according to

item

1 or 2, wherein the virtual projection plane set in the first step is a plurality of virtual projection planes.

4. In the third step, calculation of image data is performed by calculating a corresponding position on the imaging element surface from an incident angle θ with respect to the optical axis of the optical system at each pixel position on the virtual projection plane. 4. The image processing method according to claim 1, wherein image data of the pixel on the virtual projection plane is obtained from pixel data of the pixel.

5. An image processing apparatus that obtains image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving light on an imaging device having a plurality of pixels via an optical system,
The coordinates in the world coordinate system of each pixel of the virtual projection plane for which the position and size are set are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data An image processing unit that calculates image data on the virtual projection plane,
An image processing apparatus comprising:

6). An image processing apparatus that obtains image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving light on an imaging device having a plurality of pixels via an optical system,
A setting unit capable of setting the position and size of the virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data An image processing unit that calculates image data on the virtual projection plane set by the setting unit,
An image processing apparatus comprising:

7. 7. The image processing apparatus according to 6, wherein the virtual projection plane set by the setting unit is a plurality of virtual projection planes.

8. The image processing unit calculates the position of the pixel at the calculated position by calculating the corresponding position on the imaging element surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. 8. The image processing apparatus according to item 6 or 7, wherein image data at the pixel on the virtual projection plane is obtained from pixel data.

9. An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
The coordinates in the world coordinate system of each pixel of the virtual projection plane for which the position and size are set are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data Based on the image processing unit for calculating the image data on the virtual projection plane,
Program to function as.

10. An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
A setting unit capable of setting the position and size of the virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data An image processing unit for calculating image data on the virtual projection plane set by the setting unit,
Program to function as.

11. Optical system,
An imaging device having a plurality of pixels;
A setting unit for setting the position and size of the virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data And an image processing unit that calculates image data on the virtual projection plane set by the setting unit.

12 An operation unit operated by a user;
A display unit;
Have
The setting unit sets the position and size of the virtual projection plane in the world coordinate system based on an operation to the operation unit,
12. The imaging apparatus according to 11, wherein the display unit displays image data on the set virtual projection plane.

13. 13. The imaging apparatus according to 11 or 12, wherein the virtual projection plane set by the setting unit is a plurality of virtual projection planes.

14. The image processing unit calculates the position of the pixel at the calculated position by calculating the corresponding position on the imaging element surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. 14. The imaging apparatus according to any one of 11 to 13, wherein image data of the pixel on the virtual projection plane is obtained from pixel data.

According to the present invention, the virtual projection plane set in the world coordinate system is converted into the camera coordinate system using the distortion correction coefficient, and the virtual projection plane is based on the converted coordinates of the camera coordinate system and the pixel data of the image sensor. By calculating the image data, it is possible to reduce the processing time with a relatively small circuit.

It is a schematic diagram explaining the distortion correction which concerns on this embodiment. An example in which the position of the virtual projection plane VP is moved is shown. It is a block diagram which shows schematic structure of an imaging device. FIG. 4A shows a main control flow, and FIG. 4B shows a subroutine of step S20. It is a figure explaining the coordinate of the virtual projection surface VP. It is a figure which shows the correspondence of camera coordinate system xy and image pick-up element surface IA. An example in which two virtual projection planes VP are set is shown. This is an example in which the position of the virtual projection plane VP is changed with the image center o of the camera coordinate system as the rotation center. An example in which the virtual projection plane VP0 is rotated by roll is shown. An example in which the virtual projection plane VP0 is pitch rotated is shown. An example in which the virtual projection plane VP0 is rotated by yaw is shown. This is an example in which the position of the virtual projection plane VP is changed with the center ov of the virtual projection plane VP0 as the rotation center. An example in which the virtual projection plane VP0 is rotated by a virtual pitch is shown. An example in which the virtual projection plane VP0 is rotated by a virtual yaw is shown. This is an example in which the position of the virtual projection plane VP is changed with the image center o in the camera coordinate system as the movement center. An example in which the virtual projection plane VP0 is offset in the x direction is shown. An example in which the virtual projection plane VP0 is offset in the y direction is shown. An example in which the virtual projection plane VP0 is offset in the Z direction is shown. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. It is an example of a display image displayed on the display unit 120. It is an example of a display image divided and displayed on the display unit 120. It is an example of a display image divided and displayed on the display unit 120.

The present invention will be described based on an embodiment, but the present invention is not limited to the embodiment.

FIG. 1 is a schematic diagram for explaining distortion correction according to the present embodiment. In FIG. 1, X, Y, and Z are world coordinate systems, and the origin O is the lens center. Z includes the optical axis, and the XY plane includes the lens center plane LC passing through the lens center O. Point P is an object point of the object in the world coordinate system XYZ. θ is an incident angle with respect to the optical axis (coincident with the Z axis).

X and y are camera coordinate systems, and the xy plane corresponds to the image sensor surface IA. o is the center of the image and is the intersection of the optical axis Z and the image sensor surface. The point p is a point on the image sensor surface in the camera coordinate system, and the object point P is converted into the camera coordinate system using a distortion correction coefficient based on a parameter based on lens characteristics (hereinafter referred to as “lens parameter”). is there.

VP is a virtual projection plane. The virtual projection plane VP is set on the opposite side of the imaging element (and imaging element surface IA) with respect to the lens position (lens center plane LC) of the optical system. The virtual projection plane VP can be changed in position and size based on an instruction from the user to the operation unit 130 (see FIG. 3). In the present application, “position change” is a concept that includes not only the case where the virtual projection plane VP is translated on the XY plane, but also an angle change (also referred to as an attitude change) with respect to the XY plane.

In an initial state (initial position setting, the same applies hereinafter), the virtual projection plane VP is arranged at a predetermined position (Z direction) parallel to the lens center plane LC (XY direction) with a predetermined size, and the center of the virtual projection plane VP. ov is located on the Z axis. Gv is a point where the object point P is projected onto the virtual projection plane VP, and is an intersection of the object point P and a straight line passing through the lens center O and the virtual projection plane VP. A virtual projection plane VP1 in FIG. 2 shows a state in which the virtual projection plane VP0 is rotated on the XZ plane based on the input of the operation unit 130.

[Block Diagram]
FIG. 3 is a block diagram illustrating a schematic configuration of the imaging apparatus. The imaging apparatus includes an imaging unit 110, a control device 100, a display unit 120, and an operation unit 130.

The imaging unit 110 includes a lens, an imaging element, and the like. In the present embodiment, examples of the lens include a wide-angle lens and a fisheye lens.

The control device 100 includes an image processing unit 101, a setting unit 102, and a storage unit 103.

The setting unit 102 sets the position and size of the virtual projection plane VP based on an input instruction to the operation unit 130.

The image processing unit 101 creates a conversion table of each coordinate on the virtual projection plane into the camera coordinate system based on the set position and size of the virtual projection plane VP, and shoots with the imaging unit 110 using the conversion table. The processed pixel data is processed to create image data to be displayed on the display unit 120. The storage unit 103 stores a distortion correction coefficient calculated based on the lens parameters of the lens. Also, the position and size of the virtual projection plane VP and the created conversion table are stored.

The display unit 120 includes a display screen such as a liquid crystal display, and sequentially displays the image data created by the image processing unit 101 based on the pixel data captured by the imaging unit 110 on the display screen.

The operation unit 130 includes a keyboard, a mouse, or a touch panel arranged so as to be superimposed on the liquid crystal display of the display unit, and receives a user's input operation.

[Control flow]
FIG. 4 is a diagram showing a control flow of the present embodiment. FIG. 4A shows a main control flow, and FIG. 4B shows a subroutine of step S20.

In step S10 (corresponding to “first step”), the virtual projection plane VP is converted (set). For the virtual projection plane set to the initial position and size, the setting unit 102 instructs the virtual projection plane VP in the world coordinate system in response to an input instruction to the operation unit 130 as described above. The size is set. The camera viewpoint displayed on the display unit 120 is changed by changing the position as shown in FIG. 2 (corresponding to pan and tilt). Further, if the distance from the lens center O is changed with the change of the position of the virtual projection plane VP, the zoom-in / zoom-out is performed. Zooming in and zooming out can also be performed by changing the size of the virtual projection plane VP. A specific example regarding the position change of the virtual projection plane VP will be described later.

In step S10, the virtual projection plane VP is divided into the number of pixels n based on the input size setting (or the default size value). The number of pixels is preferably equal to or greater than the total number of display pixels (screen resolution) of the display unit 120. Hereinafter, as an example, both the number of pixels n and the total number of display pixels of the display unit 120 will be described under a fixed condition of 640 × 480 pixels (total number of pixels 307,000). In the present embodiment, since the interval between adjacent pixels on the virtual projection plane VP is set to be equal, the size of the virtual projection plane VP is fixed when the number of pixels n is fixed.

In step S20 (corresponding to “second and third steps”), distortion correction processing is mainly performed by the image processing unit 101 based on the state of the virtual projection plane VP set in step S10. The distortion correction process will be described with reference to FIG.

In step S21, the coordinates Gv (X, Y, Z) of the world coordinate system are acquired for each pixel Gv on the virtual projection plane VP. FIG. 5 is a schematic diagram for explaining a coordinate system. As shown in FIG. 5, point A (0, 0, Za), point B (0, 479, Zb), point C (639, 479, Zc), point D (639, 0) at the four corners of the virtual projection plane VP. , Zd) is divided into 640 × 480 pixel pixels Gv (total number of pixels: 307,000) at equal intervals, and the coordinates of all the pixels Gv in the world coordinate system are obtained.

In step S22, coordinates Gi (x, y in the corresponding camera coordinate system on the image sensor surface IA are calculated from the coordinates of the pixel Gv in the world coordinate system and the distortion correction coefficient of the imaging unit 110 stored in the storage unit 103. ) Is calculated. Specifically, the distortion correction coefficient calculated from the lens parameters of the optical system is stored in the storage unit 103, and is calculated from the incident angle θ with respect to the optical axis Z obtained from the coefficient and the coordinates of each pixel Gv. (Reference: International Publication No. 2010/032720).

FIG. 6 is a diagram showing a correspondence relationship between the camera coordinate system xy and the imaging element surface IA. In FIG. 6, points a to d are obtained by converting the points A to D in FIG. 5 into the camera coordinate system. In FIG. 5, the virtual projection plane VP surrounded by the points A to D is a rectangular plane. In FIG. 6, the area surrounded by the points a to d after the coordinate conversion to the camera coordinate system is (the virtual projection plane VP The shape is distorted (corresponding to the position). The figure shows an example of distortion in a barrel shape, but the case may be a pincushion type or a Jinkasa type (a shape that changes into a barrel type at the center and straight or pincushion at the end) due to the characteristics of the optical system. There is also.

In step S23, the pixel of the image sensor to be referenced is determined from the coordinates Gi (x ′, y ′) in the camera coordinate system. Note that x and y in the coordinates (x, y) of each pixel of the image sensor are integers, but x ′ and y ′ in the coordinates Gi (x ′, y ′) calculated in step S22 are not necessarily integers. Can take a real value with a fractional part. When x ′ and y ′ are integers as in the former and the coordinates Gi (x ′, y ′) coincide with the pixel position of the image sensor, the pixel data of the pixel of the corresponding image sensor is used as the virtual projection plane. Used as pixel data of the pixel Gv (X, Y, Z) on the VP. As in the latter case, when x ′ and y ′ are not integers and x ′ and y ′ do not match x and y, the calculated coordinates Gi (x ′, y ′) are used as pixel data of the pixel Gv. By using the pixel data of the surrounding pixels, for example, the top four pixels close to the position of the coordinate Gi (x ′, y ′), these simple average values or the distance to the coordinate Gi (x ′, y ′) Pixel data calculated by weighting four adjacent pixels may be used. The peripheral locations are not limited to 4 locations, and may be 1 location, 16 locations or more.

Steps S21 to S23 are moved from the point A (0, 0, Za), which is the starting point of FIG. 5, by one pixel (pixel) at a time to each pixel (up to the lower right end point C (639, 479, Zc)). (Pixel), image data in which distortion correction has been performed for all pixels can be acquired. This is the control related to the subroutine of step S20 shown in FIG.

Returning to the explanation of the control flow in FIG. In step S40 (corresponding to “fourth step”), the image data acquired in step S20 is displayed on the display screen of the display unit 120. Steps S20 and S40 are sequentially performed, and the image data after distortion processing based on the captured pixel data is displayed on the display unit 120 in real time.

According to the present embodiment, by setting the position and size of the virtual projection plane VP, it is possible to handle all processes including panning, tilting, and zooming processes including distortion correction processes in a batch process. The processing becomes lighter, and the processing time can be shortened with a relatively small circuit. As a result, even a relatively small circuit configuration such as ASIC (Application Specific Integrated Circuit) can be processed in real time.

[Second Embodiment]
FIG. 7 is a schematic diagram for explaining distortion correction according to the second embodiment. In the description of FIGS. 1 to 6, the single virtual projection plane is used. However, the virtual projection plane is not limited to this and may be two or more. The image data obtained at the positions of the respective virtual projection planes are switched and displayed by time division or a user instruction, or the display screen is divided and displayed at the same time. The second embodiment shown in FIG. 7 is an example in which two virtual projection planes are set. Except for the configuration shown in FIG. 7, it is the same as the embodiment described in FIGS.

FIG. 7 shows an example in which two virtual projection planes VPh and VPj are set. Both can set the position and size independently. In the drawing, the ranges corresponding to the virtual projection planes VPh and VPj on the image sensor surface IA are the areas h and j, and the points corresponding to the object points P1 and P2 are the points p1 and p2.

Image data is calculated by the flow shown in FIG. 4B for each of the set virtual projection planes VPh and VPj, and each of them is individually displayed on the display unit 120 by the process of step S40 of FIG. 4A. Displayed on the screen.

[Specific Example of Changing Position of Virtual Projection Plane VP]
8 to 11 are examples in which the position of the virtual projection plane VP0 is changed with the image center o of the camera coordinate system as the rotation center (or the movement center). As shown in FIG. 8, rotation about the x-axis with the image center o as the rotation center is pitch (also referred to as tilt), rotation about the y-axis is yaw (also referred to as pan), and rotation about the Z-axis. Is roll.

9, FIG. 10, and FIG. 11 show examples in which the virtual projection plane VP0 is rotated by roll, pitch, and yaw, respectively, based on the input set value of the rotation amount. In these figures (and also thereafter), Ca0 and Ca1 are virtual cameras, and cav is the camera viewpoint of the virtual camera Ca1 after the position change. Note that the camera viewpoint cav of the virtual camera Ca0 before the position change coincides with the Z axis.

In FIG. 9, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating it. The camera viewpoint cav coincides with before and after the position change.

In FIG. 10, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and performing the pitch rotation, and the camera viewpoint cav moves in the direction of looking up in FIG.

In FIG. 11, the virtual projection plane VP0 is obtained by changing the position of the virtual projection plane VP0 and rotating the yaw rotation, which is the virtual projection plane VP1, and the camera viewpoint cav is rotating clockwise in FIG.

12 to 14 are examples in which the position of the virtual projection plane VP is changed based on the center ov of the virtual projection plane VP0 based on the input rotation amount setting value. In the following, viewpoint conversion corresponding to rotating or changing the position of the virtual camera Ca0 is performed.

As shown in FIG. 12, one of the two axes that are orthogonal to each other on the virtual projection plane VP0 is set as Yaw-axis, and the other is set as P-axis. Both are axes passing through the center ov, and rotation around the Yaw-axis with the center ov as the rotation center is called virtual yaw rotation, and rotation around the P-axis is called virtual pitch rotation. In the initial state, the virtual projection plane VP0 is parallel to the xy plane of the camera coordinate system, and the center ov exists on the Z axis. In this initial state, P-axis is parallel to the x-axis and Yaw-axis is parallel to the y-axis.

In FIG. 13, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating the virtual pitch, and after the position change, the virtual camera Ca1 is positioned upward and the camera viewpoint cav is in a direction to look down.

14, the virtual projection plane VP1 is obtained by changing the position of the virtual projection plane VP0 and rotating the virtual yaw, and the virtual camera Ca1 and the camera viewpoint cav rotate counterclockwise after the position change.

15 to 18 are examples in which the position of the virtual projection plane VP is changed with the image center o in the camera coordinate system as the movement center. In these examples, viewpoint conversion corresponding to the parallel movement of the virtual camera Ca0 together with the virtual projection plane VP is performed.

FIGS. 16, 17, and 18 show examples in which the virtual projection plane VP0 is offset (translated) in the X, Y, and Z directions based on the input offset movement amount setting value. is there. In the initial state, the offset movement in the Z direction is the same movement as zooming in and zooming out. Except in the initial state, the offset movement in each direction is effective when moving a dark part (see an example described later) outside the imaging area of the optical system outside the image area.

[Example]
An example in which distortion correction processing is performed by the imaging apparatus of the present embodiment will be described. 19 to 26 are examples of display images displayed on the display unit 120. FIG. FIG. 19 shows an example of a distorted image when the distortion correction processing is not performed. 20 to 27 are examples of display images that have been subjected to distortion correction processing. 26 and 27 are examples in which a plurality of virtual projection planes VP are set and images corresponding to the respective virtual projection planes VP are displayed on the display unit 120.

FIG. 20 shows an example in which the virtual projection plane VP is set parallel to the lens center plane LC, and the center of the virtual projection plane VP is substantially coincident with the optical axis. In the example of FIG. 8 and the like, the display image corresponds to the virtual projection plane VP0 in the initial state. FIG. 21 shows an example in which the virtual projection plane VP0 is offset in the Z direction.

FIG. 22 shows the position of the virtual projection plane VP0 rotated by yaw (corresponding to FIG. 11). In the figure, the right end is outside the imaging area and is therefore a dark part.

FIG. 23 shows the position of the virtual projection plane VP0 rotated by pitch (corresponding to FIG. 10). Also in the figure, a dark portion is generated at the lower end.

FIG. 24 shows the virtual projection plane VP0 rotated by roll (corresponding to FIG. 9).

FIG. 25 shows the position of the virtual projection plane VP0 rotated by a virtual yaw (see FIG. 14; however, the direction of rotation is opposite to that in FIG. 14). Also in the figure, a dark part is generated at the right end.

FIG. 26 is an example in which two virtual projection planes VP are set, and FIG. 27 is an example in which three virtual projection planes VP are set, and display images corresponding to the respective virtual projection planes VP are displayed on the display unit 120. It is divided and displayed. As shown in the figure, each of the plurality of virtual projection planes VP can be rotated and moved independently. In FIG. 27, the center image and the two images on both sides differ in the offset amount in the Z direction. Is enlarged. In addition, the center image and the images on both sides overlap a part of the shooting area.

In the embodiment described above, the optical system including a single lens is exemplified as the optical system including the condensing lens. However, the condensing lens may be composed of a plurality of lenses. Of course, an optical element other than the condenser lens may be provided, and the invention of the present application can be applied. In that case, the distortion correction coefficient may be a value for the entire optical system. However, in terms of downsizing and cost reduction of the imaging device, it is most preferable to use a single lens as exemplified in the above embodiment.

DESCRIPTION OF SYMBOLS 100 Control apparatus 101 Image processing part 102 Setting part 103 Storage part 110 Imaging unit 120 Display part 130 Operation part VP Virtual projection surface LC Lens center plane IA Image sensor surface O Lens center o Image center

Claims

In an image processing method for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A first step of setting the position and size of the virtual projection plane in the world coordinate system based on a user instruction;
A second step of converting coordinates in the world coordinate system of each pixel of the virtual projection plane set in the first step into a camera coordinate system using a distortion correction coefficient of the optical system;
A third step of calculating image data of the virtual projection plane set in the first step based on the plurality of pixel data and the coordinates in the camera coordinate system converted in the second step;
An image processing method comprising:
The image processing method according to claim 1, further comprising a fourth step of displaying the image data calculated in the third step on a display unit.
3. The image processing method according to claim 1, wherein the virtual projection plane set in the first step is a plurality of virtual projection planes.
In the third step, calculation of image data is performed by calculating a corresponding position on the imaging element surface from an incident angle θ with respect to the optical axis of the optical system at each pixel position on the virtual projection plane. The image processing method according to claim 1, wherein image data of the pixel on the virtual projection plane is obtained from pixel data of the pixel.
An image processing apparatus that obtains image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving light on an imaging device having a plurality of pixels via an optical system,
The coordinates in the world coordinate system of each pixel of the virtual projection plane for which the position and size are set are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data An image processing unit that calculates image data on the virtual projection plane,
An image processing apparatus comprising:
An image processing apparatus that obtains image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving light on an imaging device having a plurality of pixels via an optical system,
A setting unit capable of setting the position and size of the virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data An image processing unit that calculates image data on the virtual projection plane set by the setting unit, and
An image processing apparatus comprising:
The image processing apparatus according to claim 6, wherein the virtual projection plane set by the setting unit is a plurality of virtual projection planes.
The image processing unit calculates the position of the pixel at the calculated position by calculating the corresponding position on the imaging element surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. The image processing apparatus according to claim 6, wherein image data at the pixels on the virtual projection plane is obtained from pixel data.
An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
The coordinates in the world coordinate system of each pixel of the virtual projection plane for which the position and size are set are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data Based on the image processing unit for calculating the image data on the virtual projection plane,
Program to function as.
An image processing apparatus program for obtaining image data subjected to distortion correction processing using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the computer comprising:
A setting unit capable of setting the position and size of the virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data An image processing unit for calculating image data on the virtual projection plane set by the setting unit,
Program to function as.
Optical system,
An imaging device having a plurality of pixels;
A setting unit for setting the position and size of the virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set by the setting unit are converted into the camera coordinate system using the distortion correction coefficient of the optical system, the coordinates converted into the camera coordinate system and the plurality of pixel data And an image processing unit that calculates image data on the virtual projection plane set by the setting unit.
An operation unit operated by a user;
A display unit;
Have
The setting unit sets the position and size of the virtual projection plane in the world coordinate system based on an operation to the operation unit,
The imaging apparatus according to claim 11, wherein the display unit displays image data on the set virtual projection plane.
The imaging apparatus according to claim 11 or 12, wherein the virtual projection plane set by the setting unit is a plurality of virtual projection planes.
The image processing unit calculates the position of the pixel at the calculated position by calculating the corresponding position on the imaging element surface from the incident angle θ with respect to the optical axis of the optical system at each position on the virtual projection plane. The image pickup apparatus according to any one of claims 11 to 13, wherein image data of the pixel on the virtual projection plane is obtained from pixel data.