JP5378143B2 - Image conversion apparatus and operation support system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image converting device that simultaneously achieves the real time property of visual point conversion processing and the resource saving property thereof. <P>SOLUTION: The image converting device (100) for generating a visual point conversion image from an original image acquired by use of a camera (2) includes: an attitude displacement detection unit (3) for detecting the attitude displacement of the camera (2); a conversion table generation means (10) for generating a conversion table (40) that associates a coordinate on the original image with a coordinate on the visual point conversion image; a conversion table storage means (12) for storing conversion tables in which the displacement to respective current attitudes of the camera (2) corresponds to the attitudes which satisfy a predetermined condition in a conversion table storage unit (4); a conversion table erasing means (13) for erasing conversion tables in which the displacement corresponds to respective attitudes that do not satisfy the predetermined condition from the conversion table storage unit (4); and a visual point conversion means (14) for subjecting the original image to a visual point conversion by use of the attitude displacement of the camera (2) and the conversion table (40). <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image conversion apparatus that performs viewpoint conversion on an original image captured by a camera, and more particularly to an image conversion apparatus that achieves both real-time viewpoint conversion processing and resource saving of the apparatus.

  2. Description of the Related Art Conventionally, a vehicle peripheral image processing apparatus that generates a bird's-eye view image based on an original image taken by an in-vehicle camera is known (for example, see Patent Document 1).

  This vehicle periphery image processing device calculates the depression angle of the in-vehicle camera in real time based on the position of the road projection point (red point) in the original image of the laser light emitted by the laser light emitting device built in the in-vehicle camera. The bird's-eye view image is generated in real time using the calculated depression angle and the conversion formula for converting the two-dimensional coordinates on the original image into the two-dimensional coordinates on the bird's-eye view image.

  There is also known an in-vehicle image processing apparatus that converts an original image captured by the in-vehicle camera into a bird's-eye image using a conversion table for converting coordinates on the original image captured by the in-vehicle camera into coordinates on the bird's-eye view image. (For example, refer to Patent Document 2).

  This in-vehicle image processing apparatus prepares in a memory a plurality of conversion tables corresponding to angles different from the standard depression angle of the in-vehicle camera by several commas, and sets the current depression angle of the in-vehicle camera detected by the angle sensor. The original image captured by the in-vehicle camera is converted into a bird's eye image while selecting a corresponding conversion table in real time.

  In this way, the vehicle periphery image processing device and the vehicle-mounted image processing device can continuously display a bird's-eye view image without any disturbance even if the depression angle of the vehicle-mounted camera changes depending on the state of the vehicle.

JP 2003-274394 A JP 2004-64441 A

  However, the vehicle peripheral image processing device described in Patent Document 1 performs processing with high calculation costs in real time, such as calculation of the depression angle of the in-vehicle camera and conversion using a conversion formula from coordinates on the original image to coordinates on the bird's-eye view image. Therefore, when the process is executed by a device with reduced specifications, there is a possibility that a display lag is generated and the bird's-eye view image cannot be displayed smoothly.

  Moreover, since the in-vehicle image processing apparatus described in Patent Document 2 stores all conversion tables in a memory in advance, it is necessary to execute conversion in real time from a coordinate on the original image to a coordinate on the bird's-eye view image. However, it is necessary to prepare a large-capacity memory that can store a large number of conversion tables necessary to cope with a wide range of depression angles, which increases the manufacturing cost of the apparatus.

  In view of the above points, an object of the present invention is to provide an image conversion apparatus that achieves both real-time performance of viewpoint conversion processing and resource saving of the apparatus, and an operation support system using the image conversion apparatus.

  In order to achieve the above-described object, an image conversion apparatus according to an embodiment of the present invention is an image conversion apparatus that generates a viewpoint conversion image from an original image acquired by a camera, and detects a displacement of the posture of the camera. A posture table, a conversion table generating unit that generates a conversion table that associates the coordinates on the original image with the coordinates on the viewpoint conversion image, and a conversion table generated by the conversion table generating unit. Conversion table storage means for storing in a conversion table storage section a conversion table corresponding to each of the postures whose displacement with respect to the current posture satisfies a predetermined condition, and the displacement of the camera with respect to the current posture does not satisfy the predetermined condition A conversion table erasing unit for erasing a conversion table corresponding to each posture from the conversion table storage unit, and the posture displacement detecting unit Conversion table or the conversion table generating means out displacement to be in the conversion table storage unit, characterized in that it comprises, and viewpoint converting means for perspective transforming said original image by using the conversion table is generated. Thereby, the image conversion apparatus according to the present embodiment performs real-time viewpoint conversion processing by preparing a conversion table in the conversion table storage unit in advance while preventing disturbance of the viewpoint conversion image due to camera posture displacement. And the resource saving performance of the apparatus by adopting a conversion table storage unit having a necessary and sufficient capacity that does not hold an extra conversion table.

  Further, the posture displacement detection unit is configured to displace an angle formed by the optical axis of the camera and a horizontal plane, a displacement of a rotation angle of the optical axis around a vertical axis passing through the camera, and the around the optical axis. It is preferable to detect at least one of the displacements of the rotation angle of the camera. Thereby, the image conversion apparatus according to the present embodiment can cope with various camera posture displacements.

  The image conversion apparatus according to the embodiment of the present invention further includes posture displacement prediction means for predicting the displacement of the camera posture, and the predetermined condition is determined based on a prediction result of the posture displacement prediction means. Is preferred. Thereby, the image conversion apparatus according to the present embodiment can flexibly determine the conversion table to be prepared in the conversion table storage unit according to the movement of the moving body and the change in the road surface state.

  An operation support system according to an embodiment of the present invention is an operation support system that supports movement or operation of an object to be operated, and moves or operates the image conversion apparatus as described above and the object to be operated. And a display unit that displays a viewpoint conversion image generated by the image conversion device. Thereby, the operation support system according to the present embodiment achieves both real-time viewpoint conversion processing and resource saving of the image conversion apparatus, while preventing disturbance of the viewpoint conversion image due to displacement of the camera posture. The movement or operation of the operated object can be supported.

  With the above-described means, the present invention can provide an image conversion apparatus that achieves both real-time performance of viewpoint conversion processing and resource saving of the apparatus, and an operation support system using the image conversion apparatus.

1 is a block diagram schematically showing a configuration example of an image conversion apparatus according to the present invention. It is a figure for demonstrating attitude | position displacement amount. It is FIG. (1) which shows typically transition of the conversion table stored in a conversion table memory | storage part. It is FIG. (2) which shows typically transition of the conversion table stored in a conversion table memory | storage part. It is a flowchart which shows the flow of a viewpoint conversion process. It is a flowchart which shows the flow of a conversion table memory | storage part update process.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically illustrating a configuration example 100 of an image conversion apparatus according to the present invention. The image conversion apparatus 100 includes, for example, a control unit 1, a camera 2, a posture displacement detection unit 3, and a conversion table storage unit. 4 includes a moving body state detection unit 5, a road surface state detection unit 6, and a display unit 7.

  The image conversion apparatus 100 is, for example, an apparatus mounted on a construction machine. The original image acquired by the camera 2 attached to the construction machine is converted from the two-dimensional coordinates on the original image to the two-dimensional coordinates on the bird's-eye view image. It is an apparatus for converting into a bird's-eye view image using a conversion table for conversion.

  Further, the image conversion apparatus 100 is, for example, a conversion table that is likely to be used next (for example, among a plurality of conversion tables that can correspond to each of various depression angles formed by the optical axis of the camera 2 (for example, This is a conversion table corresponding to each of the angle groups in which the angle difference with respect to the current value of the depression angle is less than the threshold value) and stored in the memory, and the depression angle is determined according to the inclination of the construction machine or the ground. Even when there is a change, the original image can be converted into a bird's-eye view image without delay while referring to the conversion table corresponding to the changed depression angle value from the memory.

  Next, each component of the image conversion apparatus 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), and the like. Programs corresponding to the generation means 10, the attitude displacement prediction means 11, the conversion table storage means 12, the conversion table erasure means 13, and the viewpoint conversion means 14 are stored in a ROM or NVRAM, and each RAM is used as a temporary storage area. The CPU executes a process corresponding to the means.

  The camera 2 is a device for acquiring an original image around the construction machine. For example, the camera 2 is attached to the side surface or the rear surface of the upper swing body of the construction machine so as to photograph a region that is a blind spot of the driver in the cabin. There are multiple digital cameras. In addition, the camera 2 may be attached to the front surface of the upper turning body, and may be equipped with a wide-angle lens or a fish-eye lens so that a wide range can be photographed.

  The camera 2 acquires an original image around the construction machine according to a control signal from the control unit 1, and outputs the acquired original image to the control unit 1. In addition, when the camera 2 acquires an original image using a fish-eye lens or a wide-angle lens, the camera 2 outputs a corrected original image corrected for distortion and tilt caused by using these lenses to the control unit 1. Also good.

  The posture displacement detection unit 3 is a device for detecting the posture of the camera 2, and is a device for detecting a relative posture displacement amount of the camera 2, such as a gyro sensor or an acceleration sensor, and preferably Is built in the camera 2.

  FIG. 2 is a diagram for explaining the posture displacement amount, where the X axis is a horizontal axis extending in a predetermined direction on the horizontal plane, the Z axis is a vertical axis extending perpendicular to the X axis and the horizontal plane, The Y axis is a horizontal axis that extends perpendicular to the X axis and the Z axis on a horizontal plane. The camera 2 has, for example, a state in which its optical axis OA extends along the X axis as its reference posture. In FIG. 2, the origin O is shown as a fixed point. However, in actuality, the origin O is a movable point that moves along with the movement of the construction machine, that is, the movement of the camera 2. Is also a coordinate system that moves with the movement of the origin O. The origin O preferably corresponds to a reference point (for example, the rotation center of the triaxial acceleration sensor) of the posture displacement detection unit 3.

  2A shows a rotation angle θ around the Y axis of the current optical axis OA with respect to the optical axis OA of the camera 2 in the reference posture (corresponding to the pitch angle of the construction machine), and FIG. B) shows a rotation angle ψ about the Z axis of the current optical axis OA (corresponding to the yaw angle of the construction machine) with respect to the optical axis OA of the camera 2 in the reference posture, and FIG. The rotation angle φ around the optical axis OA of the current camera 2 with respect to the camera 2 in the posture (corresponding to the roll angle of the construction machine) is shown.

  The posture displacement amount is, for example, an amount representing a difference between the posture of the camera 2 at the time T1 and the posture of the camera 2 at the time T2 (T2> T1), and includes the rotation angle θ, the rotation angle ψ, and the time T1 and the time T2. If the values of the rotation angle φ are θ1, θ2, ψ1, ψ2, φ1, and φ2, respectively, the displacement Δθ (= θ2-θ1) of the rotation angle θ, the displacement Δψ (= ψ2-ψ1) of the rotation angle ψ, and It is represented by at least one of displacements Δφ (= φ2−φ1) of the rotation angle φ.

  As described above, the amount of posture displacement is projected on the horizontal plane by the angle formed by the optical axis OA of the camera 2 at the time T1 and the optical axis OA of the camera 2 at the time T2 on the horizontal plane (horizontal projection angle). It is represented by at least one of an angle (vertical projection angle) and a rotation angle of the camera 2 around the optical axis OA generated between the time T1 and the time T2.

  The conversion table storage unit 4 is a device for storing a conversion table 40 corresponding to each posture of the camera 2 and is, for example, a volatile semiconductor memory capable of rewriting data at high speed.

  The conversion table 40 is a correspondence map representing the correspondence between the two-dimensional coordinates on the original image acquired by the camera 2 and the two-dimensional coordinates on the viewpoint conversion image. The coordinate points may correspond one-to-one, or may correspond one-to-many, many-to-one, or many-to-many.

  Further, the conversion table 40 does not necessarily have to associate all of the coordinate points on the original image with any of the coordinate points on the viewpoint conversion image.

  The conversion table 40 corresponds to each of the other postures of the camera 2 represented by one or a plurality of posture displacement amounts with respect to the current posture of the camera 2, and is converted by the conversion table storage unit 12 of the control unit 1. It can be added to the conversion table storage unit 4 or can be deleted from the conversion table storage unit 4 by the conversion table deletion means 13 of the control unit 1.

  The moving body state detection unit 5 is a device for detecting the state of the moving body. For example, the moving body state detection unit 5 is a speed sensor mounted on the construction machine for detecting the traveling speed of the construction machine, and the turning angle of the upper turning body. Detecting the amount of operation of a turning angle sensor for detecting, an inclination sensor for detecting the pitch angle or roll angle of the upper turning body, or an operating lever for operating a bucket, arm, boom, or crawler A lever operation amount detector and the like are included.

  The road surface state detection unit 6 is a means for detecting the road surface state in the traveling direction of the construction machine. For example, the road surface state detecting unit 6 is a millimeter wave radar, a laser radar, an ultrasonic radar, an image sensor, etc. Detect corners and irregularities.

  The moving body state detection unit 5 and the road surface state detection unit 6 output the detection result to the control unit 1 so that the control unit 1 can predict the displacement of the posture of the camera 2 in advance.

  The display unit 7 is a device for displaying image information, and is, for example, a liquid crystal display or a projector installed in a cabin of a construction machine, and displays a bird's-eye image output from the control unit 1.

  Next, various units included in the control unit 1 will be described.

  The conversion table generation unit 10 is a unit for generating a conversion table. For example, the conversion table generation unit 10 uses two-dimensional coordinates (pixel positions) in an original image acquired by the camera 2 using a known perspective conversion technique or reverse perspective conversion technique. A conversion table is generated by calculating two-dimensional coordinates (pixel positions) in a viewpoint conversion image that is an image viewed from a desired virtual viewpoint corresponding to each, and collecting the corresponding relationships in a correspondence map.

  The posture displacement prediction means 11 is a means for predicting the displacement of the posture of the camera 2. For example, based on the output of the moving body state detection unit 5 or the road surface state detection unit 6, how fast the construction machine is Whether it is going uphill or downhill, how much the left or right crawler rides up or gets stuck in a dent, or the construction machine inclination caused by the operation of attachments such as buckets, arms, or booms It is determined whether or not a reaction (for example, jack-up) is about to occur, and the next posture (hereinafter, “predicted posture”) taken by the camera 2 is predicted. This is because the control unit 1 can recognize in advance a conversion table that is likely to be used next based on the prediction result.

  The conversion table storage unit 12 is a unit for storing the conversion table in the conversion table storage unit 4. For example, one or a plurality of conversion table storage units 12 corresponding to a neighboring posture in which the posture displacement amount with respect to the current posture of the camera 2 is less than a predetermined value. These conversion tables are generated by the conversion table generation means 10 before the posture of the camera 2 is actually in the vicinity thereof, and the generated conversion tables are stored in the conversion table storage unit 4.

  Also, the conversion table storage unit 12 performs one conversion corresponding to the predicted posture before the posture of the camera 2 actually becomes the predicted posture according to the predicted posture of the camera 2 predicted by the posture displacement prediction unit 11. The table, or a plurality of conversion tables corresponding to the predicted posture and the neighboring postures of the predicted posture, are generated in the conversion table generation means 10 and the generated conversion tables are stored in the conversion table storage unit 4. Good.

  The conversion table erasing unit 13 is a unit for erasing the conversion table stored in the conversion table storage unit 4. For example, among the conversion tables already stored in the conversion table storage unit 4 by the conversion table storage unit 12, One or a plurality of conversion tables corresponding to a posture in which the amount of posture displacement with respect to the current posture of the camera 2 is equal to or greater than a predetermined value is deleted from the conversion table storage unit 4. This is because the conversion table storage unit 12 can store in the conversion table storage unit 4 another conversion table that is likely to be used next.

  Also, the conversion table erasing unit 13 is the conversion table already stored in the conversion table storage unit 4 by the conversion table storage unit 12, and the predicted posture of the camera 2 predicted by the posture displacement prediction unit 11 and the vicinity posture of the predicted posture. One or a plurality of conversion tables that do not correspond to any of the above may be deleted from the conversion table storage unit 4.

  The viewpoint conversion unit 14 is a unit for generating a viewpoint conversion image from the original image acquired by the camera 2. For example, the conversion table corresponding to the current posture of the camera 2 is read from the conversion table storage unit 4 and referred to. However, the attribute values (for example, luminance, hue, and saturation) of each pixel in the original image acquired by the camera 2 are used as attributes of the corresponding pixel in the viewpoint conversion image that is an image viewed from a desired virtual viewpoint. A viewpoint conversion image is generated by assigning to a value. Note that the viewpoint conversion unit 14 may generate a viewpoint conversion image while performing pixel interpolation, pixel thinning, pixel synthesis, and the like as necessary.

  In addition, when the conversion table corresponding to the current posture of the camera 2 does not exist in the conversion table storage unit 4, the viewpoint conversion unit 14 causes the conversion table generation unit 10 to generate a conversion table corresponding to the current posture. The viewpoint conversion image may be generated with reference to the generated conversion table, or the conversion table stored in the conversion table storage unit 4 corresponds to the position closest to the current position. The viewpoint conversion image may be generated with reference to the conversion table.

  FIG. 3 is a diagram schematically showing the transition of the conversion table stored in the conversion table storage unit 4, and the conversion when the rotation angle θ around the Y axis of the optical axis OA of the camera 2 is adopted as the posture displacement amount. FIG. 6 is a diagram schematically showing transition (storage and deletion) of a table (for example, a conversion table for allowing disturbance of a viewpoint conversion image caused by a posture displacement of the camera 2 due to a pitch of a construction machine to be canceled in real time. Shows the transition.)

  FIG. 3A shows the state of the conversion table storage unit 4 at time T1, and is represented by a conversion table 40-3 (fine hatched rectangle) corresponding to the current posture of the camera 2 (rotation angle θ = 0). .), Conversion tables 40-2 and 40-4 (represented by rough hatched rectangles) corresponding to the primary adjacent posture (rotation angle θ = ± a) with respect to the current posture, and the current posture. It shows that conversion tables 40-1 and 40-5 (represented by rough hatched rectangles) corresponding to the secondary adjacent posture (rotation angle θ = ± 2a) exist in the conversion table storage unit 4. The primary adjacent posture means an adjacent posture directly adjacent to the current posture, and the secondary adjacent posture means an adjacent posture directly adjacent to the temporary adjacent posture. The same applies to the third and subsequent adjacent postures. The constant a is a unit angle of the rotation angle θ that is a posture displacement amount of the camera 2, and a person viewing the viewpoint conversion image notices the distortion of the viewpoint conversion image caused by the displacement (rotation) of the posture of the camera 2. This corresponds to the minimum displacement (rotation angle) that can be achieved.

  FIG. 3B shows the state of the conversion table storage unit 4 at time T2 (T2> T1). The posture of the camera 2 changes from the posture at the time T1 (rotation angle θ = 0) to the current posture (rotation angle). It shows that it was displaced to θ = + a).

  In this case, the conversion table erasing unit 13 converts the conversion table corresponding to the primary adjacent posture and the secondary adjacent posture with respect to the current posture (rotation angle θ = + a) into a conversion table necessary for dealing with the subsequent posture displacement. As the conversion table storage unit 4 in advance, the conversion table 40-1 (represented by a white rectangle surrounded by a dotted line) corresponding to one of the extraneous tertiary adjacent postures (rotation angle θ = −2a). Is deleted from the conversion table storage unit 4.

  On the other hand, the conversion table storage unit 12 is expressed by a conversion table 40-6 (rough hatched rectangle surrounded by a thick line) corresponding to one of the insufficient secondary adjoining postures (rotation angle θ = + 3a). ) Is generated by the conversion table generating means 10.

  3C shows the state of the conversion table storage unit 4 at time T3 (T3> T2), and the posture of the camera 2 changes from the posture at the time T2 (rotation angle θ = + a) to the current posture (rotation angle). It shows that it has further displaced at θ = + 2a).

  In this case, the conversion table erasing unit 13 converts the conversion table corresponding to the primary adjacent posture and the secondary adjacent posture with respect to the current posture (rotation angle θ = + 2a) into a conversion table necessary for dealing with the subsequent posture displacement. As a conversion table 40-2 (represented by a white rectangle surrounded by a dotted line) corresponding to one of the extraneous tertiary adjacent postures (rotation angle θ = −a). Is deleted from the conversion table storage unit 4.

  On the other hand, the conversion table storage means 12 is expressed by a conversion table 40-7 (rough hatched rectangle surrounded by a thick line) corresponding to one of the insufficient secondary adjoining postures (rotation angle θ = + 4a). ) Is generated by the conversion table generating means 10.

  FIG. 3D shows the state of the conversion table storage unit 4 at time T2-1 (T2-1> T1). The posture of the camera 2 is changed from the posture at time T1 (rotation angle θ = 0) to the current state. It shows that it was displaced to a posture (rotation angle θ = + 2a). Note that the state shown in FIG. 3D indicates that the amount of posture displacement is larger than that in the state shown in FIG.

  In this case, the conversion table storage means 12 considers the magnitude of the posture displacement amount, and converts the conversion table corresponding to the secondary adjacent posture and the fourth adjacent posture with respect to the current posture (rotation angle θ = + 2a). The conversion table storage unit 4 is prepared in advance as a conversion table necessary for dealing with the subsequent posture displacement.

  Therefore, the conversion table erasing unit 13 corresponds to the conversion table 40-4 corresponding to one extra primary adjacent attitude (rotation angle θ = + a) and one of the tertiary adjacent attitudes (rotation angle θ = −a). The conversion table 40-2 to be deleted is deleted from the conversion table storage unit 4.

  On the other hand, the conversion table storage unit 12 includes one of the conversion table 40-7 corresponding to one of the shortage secondary adjoining postures (rotation angle θ = + 4a) and one of the fourth adjoining postures (rotation angle θ = + 6a). The conversion table generation means 10 generates a conversion table 40-8 corresponding to the above.

  In this way, the image conversion apparatus 100 restricts the number of conversion tables prepared in advance in the conversion table storage unit 4 to a predetermined number (in this case, five), and the next posture of the camera 2. Can be prepared so that the viewpoint conversion means 14 can refer to the conversion table corresponding to the above in real time.

  4 is a diagram schematically showing the transition of the conversion table stored in the conversion table storage unit 4 as in FIG. 3. The rotation angle θ about the Y axis of the optical axis OA of the camera 2 and the light of the camera 2 are shown in FIG. It is a figure which shows typically transition of the conversion table in the case of employ | adopting rotation angle (phi) about the axis | shaft OA as an attitude | position displacement amount (For example, the viewpoint conversion image resulting from the attitude displacement of the camera 2 by the pitch and roll of a construction machine) The transition of the conversion table for canceling the disturbance of the real time is shown, since the yaw of the construction machine is a normal movement caused by the turning of the upper turning body, the posture of the camera 2 by the yaw of the construction machine The change in the viewpoint conversion image caused by the displacement may not be recognized as disturbance.)

  FIG. 4A shows the state of the conversion table storage unit 4 at time T1, and the conversion table 40-e (fine hatching) corresponding to the current posture of the camera 2 (rotation angle θ = 0, rotation angle φ = 0). And conversion tables 40-a to 40-d corresponding to the primary adjacent posture (rotation angle θ = ± a or rotation angle φ = ± b) with respect to the current posture, and 40-f to 40-i (represented by a rough hatched rectangle) are present in the conversion table storage unit 4. The constant b is a unit angle of the rotation angle φ, which is the posture displacement amount of the camera 2, as with the constant a.

  FIG. 4B shows the state of the conversion table storage unit 4 at time T2 (T2> T1). The posture of the camera 2 is changed from the posture at time T1 (rotation angle θ = 0, rotation angle φ = 0). This indicates that the current posture (rotation angle θ = + a, rotation angle φ = + b) has been displaced.

  In this case, the conversion table erasing unit 13 converts the conversion table corresponding to the primary adjacent posture with respect to the current posture (rotation angle θ = + a, rotation angle φ = + b) into the conversion necessary for dealing with the subsequent posture displacement. Since the table is prepared in advance in the conversion table storage unit 4, five extra secondary adjacent postures (rotation angle θ = −a, rotation angle φ = + b), (rotation angle θ = −a, rotation angle). (φ = 0), (rotation angle θ = −a, rotation angle φ = −b), (rotation angle θ = 0, rotation angle φ = −b), (rotation angle θ = + a, rotation angle φ = −b) Conversion tables 40-a, 40-d, and 40-g to 40-i (represented by white rectangles surrounded by dotted lines) corresponding to each of the above are deleted from the conversion table storage unit 4.

  On the other hand, the conversion table storage means 12 has five primary adjoining postures (rotation angle θ = 0, rotation angle φ = + 2b), (rotation angle θ = + a, rotation angle φ = + 2b), and (rotation angle). θ = + 2a, rotation angle φ = + 2b), (rotation angle θ = + 2a, rotation angle φ = + b), and conversion tables 40-j to 40 corresponding to (rotation angle θ = + 2a, rotation angle φ = 0), respectively. -N (represented by a rough hatched rectangle surrounded by a thick line) is generated by the conversion table generation means 10.

  In this way, the image conversion apparatus 100 is prepared in the conversion table storage unit 4 even when the posture displacement amount is expressed by a plurality of parameters (in this case, the rotation angle θ and the rotation angle φ). The viewpoint conversion means 14 is prepared so that the conversion table corresponding to the next posture of the camera 2 can be referred to in real time while limiting the number of conversion tables to a predetermined number (in this case, nine). I can keep it.

  Next, a process in which the image conversion apparatus 100 converts an original image into a viewpoint converted image (hereinafter referred to as “viewpoint conversion process”) will be described with reference to FIG. FIG. 5 is a flowchart showing the flow of the viewpoint conversion process, and the image conversion apparatus 100 repeatedly executes the viewpoint conversion process at predetermined time intervals while the camera 2 outputs the original image.

  First, the control unit 1 acquires an original image taken by the camera 2 (step S1), and based on the output of the posture displacement detection unit 3, the posture of the camera 2 at the time of the previous viewpoint conversion processing and the current viewpoint. It is determined whether or not there is a difference between the posture of the camera 2 at the time of executing the conversion process (whether the posture of the camera 2 is displaced) (step S2). Note that the control unit 1 starts the viewpoint conversion process with the posture of the camera 2 at a predetermined timing as a reference posture, and the predetermined timing is, for example, when the image conversion apparatus 100 is activated or a virtual viewpoint Is fixed when the driver inputs a predetermined input to the image conversion apparatus 100.

  When it is determined that the posture of the camera 2 is not displaced (NO in step S2), the control unit 1 uses the conversion table currently referenced by the viewpoint conversion unit 14 to store the camera. A viewpoint conversion image is generated from the original image photographed by 2 (step S3).

  On the other hand, when it is determined that the posture of the camera 2 has been displaced (YES in step S2), the control unit 1 determines whether a conversion table corresponding to the post-displacement posture exists in the conversion table storage unit 4. (Step S4).

  When it is determined that the conversion table corresponding to the post-displacement posture exists in the conversion table storage unit 4 (YES in step S4), the control unit 1 is stored in the conversion table storage unit 4 by the viewpoint conversion unit 14. A viewpoint conversion image is generated from an image captured by the camera 2 using a conversion table corresponding to the post-displacement posture (step S8).

  When it is determined that the conversion table corresponding to the post-displacement posture does not exist in the conversion table storage unit 4 (NO in step S4), the control unit 1 stores another conversion table in the conversion table storage unit 4. It is determined whether there is an empty space (step S5).

  When it is determined that there is an empty space (YES in step S5), the control unit 1 uses the conversion table generation unit 10 to generate a conversion table corresponding to the post-displacement posture (step S7), and the generated conversion table is displayed. A viewpoint conversion image is generated from the original image captured by the camera 2 using the image data (step S8).

  Thereafter, the control unit 1 stores the generated conversion table in the conversion table storage unit 4 by the conversion table storage unit 12. This is so that it can be used in subsequent viewpoint conversion processing. The control unit 1 stores the generated conversion table in the conversion table storage unit 4 and generates a viewpoint conversion image from the original image captured by the camera 2 using the stored conversion table. Good.

  On the other hand, if it is determined that there is no free space (NO in step S5), the control unit 1 uses the conversion table erasing means 13 to change the displacement that does not correspond to the posture after the displacement or the posture near the posture after the displacement. A conversion table unrelated to the subsequent posture is deleted from the conversion table storage unit 4 (step S6), and an empty space for storing another conversion table corresponding to the post-displacement posture is created.

  Thereafter, the control unit 1 uses the conversion table generation means 10 to generate a conversion table corresponding to the post-displacement posture (step S7), and converts the viewpoint from the original image captured by the camera 2 using the generated conversion table. An image is generated (step S8), and this viewpoint conversion process is terminated.

  Next, a process in which the image conversion apparatus 100 updates the contents of the conversion table storage unit 4 (hereinafter referred to as “conversion table storage unit update process”) will be described with reference to FIG. FIG. 6 is a flowchart showing the flow of the conversion table storage unit update process. The image conversion apparatus 100 stores this conversion table in the background separately from the viewpoint conversion process while the camera 2 outputs the original image. The part update process is repeatedly executed at predetermined time intervals.

  First, based on the current posture of the camera 2, the control unit 1 stores it in the conversion table storage unit 4 necessary for realizing real-time viewpoint conversion when the posture of the camera 2 is displaced later. A power conversion table is determined (step S10).

  For example, the control unit 1 determines a conversion table corresponding to a nearby posture close to the current posture of the camera 2 as a conversion table to be stored in the conversion table storage unit 4.

  The control unit 1 predicts how the posture of the camera 2 will be displaced in the future based on the output of at least one of the moving body state detection unit 5 and the road surface state detection unit 6 by the posture displacement prediction unit 11. Then, the conversion table to be stored in the conversion table storage unit 4 may be determined.

  Thereafter, the control unit 1 determines whether or not an unnecessary conversion table exists in the conversion table storage unit 4 (step S11). The unnecessary conversion table is a conversion table other than the conversion table that should be stored in the conversion table storage unit 4, and corresponds to, for example, a posture other than the current posture of the camera 2 and its neighboring posture. It is a conversion table.

  When it is determined that an unnecessary conversion table exists in the conversion table storage unit 4 (YES in step S11), the control unit 1 deletes the unnecessary conversion table from the conversion table storage unit 4 by the conversion table deletion unit 13. (Step S12). The control unit 1 predicts how the posture of the camera 2 will be displaced in the future based on the output of at least one of the moving body state detection unit 5 and the road surface state detection unit 6 by the posture displacement prediction unit 11. Then, if it is determined that a conversion table corresponding to a neighboring posture close to the current posture of the camera 2 is not necessary, even if it is a conversion table corresponding to a neighboring posture close to the current posture, The conversion table corresponding to the posture may be deleted. This is to create an empty space for storing a necessary conversion table.

  Thereafter, the control unit 1 determines whether or not all of the conversion tables deemed necessary already exist in the conversion table storage unit 4 (step S13). This is to avoid the same conversion table being generated twice.

  If all the conversion tables that are deemed necessary already exist in the conversion table storage unit 4 (YES in step S13), the control unit 1 updates the conversion table storage unit without generating a new conversion table. End the process.

  On the other hand, when all or a part of the conversion table deemed necessary does not yet exist in the conversion table storage unit 4 (NO in step S13), the control unit 1 uses the conversion table generation means 10 to make the necessity. A conversion table determined to be present is generated, and the generated conversion table is stored in the conversion table storage unit 4 by the conversion table storage unit 12 (step S14), and the conversion table storage unit update process is terminated.

  With the above configuration, the image conversion apparatus 100 prevents viewpoint distortion caused by the displacement of the posture of the camera 2 and prepares a necessary conversion table in the conversion table storage unit 4 in advance, thereby converting viewpoints. It is possible to achieve both real-time processing and resource saving of the apparatus by adopting the conversion table storage unit 4 having a necessary and sufficient capacity that does not hold an extra conversion table.

  Further, the image conversion apparatus 100 can reduce the manufacturing cost by adopting the conversion table storage unit 4 having a necessary and sufficient capacity as compared with the case where the conversion table storage unit 4 having a capacity larger than necessary is employed.

  In addition, the image conversion apparatus 100 employs a gyro sensor, an acceleration sensor, or the like as the posture displacement detection unit 3, and calculates the relative displacement between the posture of the camera 2 at one time point and the posture of the camera 2 at another time point. Since it is not necessary to always acquire the posture of the camera 2 as a displacement with respect to a predetermined reference posture while employing the absolute angle sensor, the manufacturing cost can be further reduced by employing a cheaper sensor than the absolute angle sensor. Can be reduced. Note that the image conversion apparatus 100 can adopt an absolute angle sensor as the posture displacement detection unit 3 as a matter of course.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiments, the posture displacement amount is described by taking a combination of the rotation angles θ, ψ, and φ in the three-dimensional orthogonal coordinate system as an example, but in other coordinate systems such as a three-dimensional polar coordinate system. It may be represented by a variable.

  In the above-described embodiment, the image conversion apparatus 100 generates the bird's-eye view image from the original image captured by the camera 2, but may generate a viewpoint conversion image based on any other virtual viewpoint. .

  In the above-described embodiment, the image conversion apparatus 100 generates one viewpoint conversion image from one original image captured by one camera 2, but a plurality of original images captured by a plurality of cameras 2. The images may be combined to generate one viewpoint converted image, or the plurality of original images may be combined to generate a plurality of viewpoint converted images.

  In the above-described embodiment, the image conversion device 100 is mounted with a camera on a self-propelled construction machine having movable members such as a bucket, an arm, a boom, and a turning mechanism, and presents a viewpoint conversion image to the driver. However, it is built in an operation support system that supports the movement of the construction machine and the operation of these movable members, but it is self-propelled but does not have a movable member, such as an automobile, a motorcycle or a ship. Or viewpoint conversion caused by displacement of the camera posture mounted on other operated objects including a fixed operated object that has a movable member but does not self-propell like an industrial machine or a fixed crane, etc. You may incorporate in the operation assistance system which assists the movement or operation of those other to-be-operated bodies, preventing the disorder of an image.

DESCRIPTION OF SYMBOLS 1 Control part 2 Camera 3 Posture displacement detection part 4 Conversion table memory | storage part 5 Mobile body state detection part 6 Road surface state detection part 7 Display part 10 Conversion table production | generation means 11 Posture displacement prediction means 12 Conversion table storage means 13 Conversion table deletion means 14 Viewpoint conversion means 40 Conversion table 100 Image conversion device

Claims (4)

An image conversion device that generates a viewpoint conversion image from an original image acquired by a camera,
A posture displacement detector for detecting displacement of the camera posture;
Conversion table generating means for generating a conversion table for associating coordinates on the original image with coordinates on the viewpoint conversion image;
A conversion table generated by the conversion table generation unit, and a conversion table storage unit that stores in a conversion table storage unit the conversion table corresponding to each of the postures in which the displacement of the camera with respect to the current posture satisfies a predetermined condition; ,
Conversion table erasure means for erasing the conversion table corresponding to each of the postures in which the displacement of the camera with respect to the current posture does not satisfy the predetermined condition;
Viewpoint conversion means for converting the viewpoint of the original image using the displacement detected by the posture displacement detection section and the conversion table in the conversion table storage section or the conversion table generated by the conversion table generation means;
An image conversion apparatus comprising: The posture displacement detection unit includes a displacement of an angle formed by the optical axis of the camera and a horizontal plane, a displacement of a rotation angle of the optical axis around a vertical axis passing through the camera, and a displacement of the camera around the optical axis. Detecting at least one of the displacements of the rotation angle,
The image conversion apparatus according to claim 1. A posture displacement prediction means for predicting the displacement of the posture of the camera;
The predetermined condition is determined based on a prediction result of the posture displacement prediction unit.
The image conversion apparatus according to claim 1. An operation support system that supports movement or operation of an object to be operated,
The image conversion device according to any one of claims 1 to 3,
A display unit installed in an operation room for moving or operating the object to be operated, and displaying a viewpoint conversion image generated by the image conversion device;
An operation support system comprising:

Images