WO2023157443A1

WO2023157443A1 - Object orientation calculation device and object orientation calculation method

Info

Publication number: WO2023157443A1
Application number: PCT/JP2022/045955
Authority: WO
Inventors: 鳴劉; 豪一小野
Original assignee: 株式会社日立製作所
Priority date: 2022-02-21
Filing date: 2022-12-14
Publication date: 2023-08-24

Abstract

This object orientation calculation device has: an image vertex recognition unit into which a two-dimensional image captured by a two-dimensional camera and a three-dimensional image captured by a three-dimensional camera are inputted, and which recognizes a plurality of vertices of an object according to the two-dimensional image; a three-dimensional reference point determination unit that, on the basis of coordinates for the vertices recognized by the image vertex recognition unit, determines the three-dimensional coordinates of a three-dimensional reference point; and an object orientation calculation unit that calculates the angle of one side surface of the object relative to a prescribed axis for a prescribed coordinate system. The object orientation calculation unit calculates the angle of the object on the basis of the coordinates for the vertices in three-dimensional coordinates, a constraint pertaining to the three-dimensional reference point in the three-dimensional coordinates, and a constraint pertaining to the three-dimensional coordinates that correspond to the two-dimensional coordinates of the vertices based on the characteristics of the two-dimensional image. This makes it possible to improve the accuracy of calculating the orientation of pallets on which loads are mounted, and to smoothly carry out a conveyance operation using a forklift or other cargo-handling apparatus, in a forklift transportation system at a physical distribution site.

Description

OBJECT POSITION CALCULATION DEVICE AND OBJECT POSITION CALCULATION METHOD

The present invention relates to an object orientation calculation device and an object orientation calculation method, and in particular, in distribution work in a warehouse or the like, accurate calculation of the orientation of a pallet on which cargo is loaded can be made to facilitate transportation work using cargo handling equipment such as a forklift. The present invention relates to an object orientation calculation apparatus and an object orientation calculation method suitable for performing

In the recent logistics style, it is common to use a loading platform called a pallet and a forklift to carry the cargo. The pallet has insertion openings (fork pockets) on both sides of the surface on which the cargo is to be placed. work becomes possible.

In view of this logistics work style, technologies have been proposed for image recognition of pallets, grasping their positions, transporting them with unmanned automatic forklifts, and assisting drivers of manned forklifts.

In order to recognize the image of the pallet, there is a technology that recognizes the position and orientation of the pallet using 2D images and 3D point clouds using depth cameras. In the position and orientation recognition of such an object, generally two or more depth points (for example, left vertex and right vertex) on a distant object are used for recognition. There is a problem that the measurement quality of the 3D point cloud is low due to the site environment and the state of the detection object, and it is necessary to recognize the position and orientation of the target object using a few point clouds with high measurement quality.

An algorithm for obtaining the position and orientation of an object using 3D reference points is described, for example, in Non-Patent Document 1. This non-patent document 1 describes mathematical formulas representing the positional relationship between 3D reference points and their images in the 2D plane (§2 THE CAMERA POSE FROM THREE POINTS REVISITED, Fig 1).

In relation to this, Patent Document 1 discloses a three-dimensional measurement and display apparatus for calculating the position and orientation of an object based on a depth image acquired by a depth camera and a plane area extracted from the depth image. is disclosed.

International Publication No. 2014/147863

The problem to be solved by the invention will be described below with reference to FIG.

FIG. 13 is a perspective view of a pallet and a diagram for explaining specifications.
As mentioned above, due to the measurement problem of the current depth camera, the measurement quality of the 3D point cloud is low. think of.

Therefore, we will detect the position and orientation of the pallet by reducing the necessary 3D point cloud and using the 3D 1 point + 2D 1 point on the side of the pallet that can be detected and the positional relationship between these two points.

Here, detecting the position and orientation of the pallet specifically means obtaining the tilt of the camera and the pallet involved in the measurement (the specific spatial image will be explained later).

The pallet 10, as shown in FIG. 13, has a pallet side 12 with a height h and a width s, and is provided with fork pockets 11 for inserting and carrying the forks of two forklifts, An inter-fork-pocket rectangular region 13 is formed between the left and right fork pockets 11l and 11r. There are various standards for pallets. For example, the T11 type pallet specified by JIS has s=1100 mm and h=144 mm.

When performing pallet pose detection, the bounding box is clipped by image recognition, and the recognized four vertices in 2D of its sides are a ₁₁ , a ₁₂ , a ₂₁ , a ₂₂ as shown in FIG. Suppose there is The coordinate system is assumed to be a three-dimensional orthogonal coordinate system such as (X, Y, Z) as shown in the figure.

Here, an attempt is made to detect the position and orientation of the pallet from the upper right 2D point _a22 and the 3D point corresponding to the upper left _a21 . , the coordinates of a ₁₁ become A _{11 , the coordinates of a 21 become A 21 , and the 3D point corresponding to the coordinate A 21} _of _the _2D point including the image recognition error causes an error in the position and orientation of the pallet becomes large.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an object orientation calculation apparatus and an object orientation calculation method that can accurately calculate the orientation of a pallet on which cargo is to be loaded and can smoothly carry out a transportation operation using cargo handling equipment such as a forklift. be.

The configuration of the object orientation calculation apparatus of the present invention is preferably such that a two-dimensional image taken by a two-dimensional camera and a three-dimensional image taken by a three-dimensional camera are input, and the target object is positioned with respect to a predetermined coordinate system. An object orientation calculation device for calculating an angle, comprising: an image vertex recognition unit for recognizing a plurality of vertices of a target object from a two-dimensional image; and an object orientation calculation unit for calculating an angle of one side surface of the target object with respect to a predetermined axis with respect to a predetermined coordinate system. Based on the coordinates of vertices in 3D coordinates, the constraints of 3D reference points in 3D coordinates, and the 2D coordinates of vertices and the corresponding 3D coordinates constraints based on the properties of the 2D image. to calculate the angle.

According to the present invention, it is possible to provide an object attitude calculation apparatus and an object attitude calculation method that can accurately calculate the attitude of a pallet on which cargo is to be loaded and can smoothly carry out a transportation operation using cargo handling equipment such as a forklift. can.

1 is a diagram showing a configuration of a forklift transportation system at a physical distribution site according to Embodiment 1; FIG. 2 is a hardware/software configuration diagram of an object orientation calculation device; FIG. 4 is a flowchart showing processing related to object orientation calculation of the forklift transport system; 4 is a flowchart showing object orientation calculation processing according to the first embodiment. It is the figure which showed the front surface of the pallet seen from the forklift side. It is a figure which shows the positional relationship of the 3D camera and the palette seen from the back of the 3D camera. It is a figure which shows the positional relationship of the 3D camera and the palette seen from the upper direction of the 3D camera. FIG. 10 is a diagram showing the configuration of a forklift transportation system at a physical distribution site according to Embodiment 2; 10 is a flowchart showing object orientation calculation processing according to the second embodiment. FIG. 10 is a diagram showing specifications related to 3D reference point coordinate correction processing of the second embodiment on the front surface of the pallet; It is a figure which shows an example of a center point correction data table. FIG. 10 is a diagram showing specifications related to 3D reference point coordinate correction processing of Embodiment 3 on the front surface of the pallet; It is a perspective view of a pallet and a figure explaining specifications.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are exemplifications for explaining the present invention, and are appropriately omitted and simplified for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. in order to facilitate the understanding of the invention. As such, the present invention is not necessarily limited to the locations, sizes, shapes, extents, etc., disclosed in the drawings.

As examples of various types of information, expressions such as "table", "list", and "queue" may be used, but various types of information may be expressed in data structures other than these. For example, various information such as "XX table", "XX list", and "XX queue" may be referred to as "XX information". When describing identification information, expressions such as “identification information”, “identifier”, “name”, “ID”, and “number” are used, but these can be replaced with each other.

When there are multiple components that have the same or similar functions, they may be described with the same reference numerals with different suffixes. Further, when there is no need to distinguish between these constituent elements, the subscripts may be omitted in the description.

In the examples, the processing performed by executing the program may be explained. Here, the computer executes a program by means of a processor (eg, CPU, GPU) and performs processing determined by the program while using storage resources (eg, memory) and interface devices (eg, communication port). Therefore, the main body of the processing performed by executing the program may be the processor. Similarly, a main body of processing executed by executing a program may be a controller having a processor, a device, a system, a computer, or a node. The subject of the processing performed by executing the program may be an arithmetic unit, and may include a dedicated circuit for performing specific processing. Here, the dedicated circuit is, for example, FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD (Complex Programmable Logic Device), or the like.

The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the distribution target program, and the processor of the program distribution server may distribute the distribution target program to other computers. Also, in the embodiment, two or more programs may be implemented as one program, and one program may be implemented as two or more programs.

Each embodiment according to the present invention will be described below with reference to FIGS. 1 to 12. FIG.

[Embodiment 1]
A first embodiment according to the present invention will be described below with reference to FIGS. 1 to 7. FIG.

First, the configuration of a forklift transportation system at a distribution site will be described using FIGS. 1 and 2.

The forklift 20 is a device that carries by inserting the forks 25 into the fork pockets 11 of the pallet 10 . The forklift 20 has an operation command unit 30 that gives instructions to the driving wheels 28 and the fork operation unit 26 to operate them according to an external command or an operation program.

A 2D (Dimension) camera (RGB camera) 21 captures a two-dimensional image (2D point image), and a 3D (Dimension) camera (Depth camera) 22 captures a three-dimensional image (3D point image). image) is captured and wirelessly transmitted to the access point 5 via the external interface unit 27 . The 3D camera 22 is a camera that can refer to information about the depth of the captured image. The 3D camera 22 may be realized by a stereo system that measures the distance with two cameras, may be a ToF (Time of Flight) system that measures the reflection time of light, or may be a system that emits special light. It may be a structured lighting scheme that measures depth from the state.

Then, the image captured from the access point 5 is transmitted to the object orientation calculation device 100.

Also, the radar sensor 40 measures surrounding objects and transmits the measured data to the operation command unit 30 .

The object orientation calculation device 100 is a device that obtains the orientation of the pallet 10 (the angle formed with the coordinate system) based on the captured image. The object orientation calculation apparatus 100 includes functional units such as an image vertex recognition unit 101, a 3D (dimensional) reference point determination unit 102, an object angle calculation unit 103, an image reception unit 104, an object angle transmission unit 105, and a storage unit 110. have.

The image vertex recognition unit 101 is a functional unit that recognizes the vertices of the sides of the pallet 10 by performing image recognition from a 2D point image. The 3D reference point determination unit 102 is a functional unit that determines a 3D reference point (details will be described later) used for object angle calculation from points in the 3D point image. The object angle calculator 103 is a functional unit that calculates the angle of the pallet 10 from 2D points and 3D points. The image receiving unit 104 is a functional unit that receives a two-dimensional image (2D point image) captured by the 2D camera 21 and a three-dimensional image captured by the 3D camera 22 . The object angle transmission unit 105 is a functional unit that transmits the calculated angle of the pallet 10 .

The operation command unit 30 of the forklift 20 receives the angle of the pallet 10 via the access point 5 and the external interface unit 27, and generates appropriate pallet operation data.

The storage unit 110 is a functional unit that holds data necessary for the object orientation calculation apparatus 100, and includes an image data DB 111 that stores 2D image data and 3D image data, and a measurement data DB 112 that stores measurement data necessary for operation. , holds an object orientation calculation data DB 113 that stores data for calculating the orientation of an object.

Next, the hardware/software configuration of the object orientation calculation apparatus 100 will be described with reference to FIG.
As a hardware configuration of the object orientation calculation apparatus 100, for example, it is realized by a general information processing apparatus such as a personal computer shown in FIG.

The object orientation calculation device 100 includes a CPU (Central Processing Unit) 202, a main memory device 204, a network I/F (InterFace) 206, a display I/F 208, an input/output I/F 210, and an auxiliary memory I/F 212, which are coupled via a bus. It is in the form of

The CPU 202 controls each part of the object orientation calculation device 100, loads necessary programs into the main storage device 204, and executes them.

The main storage device 204 is normally composed of a volatile memory such as a RAM, and stores programs executed by the CPU 202 and data to be referred to.

A network I/F 206 is an interface for connecting to a network.

A display I/F 208 is an interface for connecting a display device 220 such as an LCD (Liquid Crystal Display).

The input/output I/F 210 is an interface for connecting input/output devices. In the example of FIG. 2, a keyboard 230 and a mouse 232 as a pointing device are connected.

The auxiliary storage I/F 212 is an interface for connecting auxiliary storage devices such as HDD (Hard Disk Drive) 250 and SSD (Solid State Drive).

The HDD 250 has a large storage capacity and stores programs for executing this embodiment. An image vertex recognition program 261, a 3D point determination program 262, an object angle calculation program 263, an image reception program 264, and an object angle transmission program 265 are installed in the object orientation calculation apparatus 100. FIG.

The image vertex recognition program 261, the 3D point determination program 262, the object angle calculation program 263, the image reception program 264, and the object angle transmission program 265 are the image vertex recognition unit 101, the 3D reference point determination unit 102, the object angle calculation unit 103, It is a program that executes the functions of the image reception unit 104 and the object angle transmission unit 105 .

The HDD 250 also holds an image data DB 111, a measurement data DB 112 of measurement data necessary for operation, and an object orientation calculation data DB 113 used to calculate the orientation of an object (pallet).

Next, the processing related to the calculation of the object orientation of the forklift transportation system will be described using FIGS. 3 and 4. FIG.

First, the object orientation calculation device 100 receives a two-dimensional image captured by the 2D camera 21 via the access point 5 (S01).

Next, the object orientation calculation device 100 receives the three-dimensional image captured by the 3D camera 22 via the access point 5 (S02).

Next, the object orientation calculation device 100 calculates the orientation of the object (pallet) based on the two-dimensional image captured by the 2D camera 21 and the three-dimensional image captured by the 3D camera 22 (S03). This process will be described in detail below with reference to FIG.

Next, the object orientation calculation device 100 transmits information (pallet angle) on the calculated orientation of the object (pallet) to the forklift via the access point 5 (S04).

Next, the details of the object orientation calculation processing will be described using FIGS. 13 and 4 already described.

This process corresponds to S03 in FIG.

First, the image vertex recognition unit 101 of the object orientation calculation device 100 performs image recognition processing by AI learning according to the pallet model, cuts out a bounding box from the two-dimensional image, and extracts four vertices of the pallet viewed from the forklift 20 side (Fig. 13 ₍ _S10 ₎ _. However, in practice, due to errors, A ₁₁ , A ₁₂ , A ₂₁ , and A ₂₂ corresponding to a ₁₁ , a ₁₂ , a ₂₁ , and a ₂₂ are obtained for the four vertices, respectively.

Next, the 3D reference point determining unit 102 of the object orientation calculation apparatus 100 selects a reference point for a 3D point (hereinafter referred to as a "3D (Dimension) reference point") from the three-dimensional image captured by the 3D camera 22. is calculated (S20). A method for obtaining the coordinates of the 3D reference point will be described in detail later.

The posture of the object (pallet) is calculated from the 2D coordinates of one of the four vertices seen from the forklift 20 side and the 3D reference point (S30). A detailed algorithm for calculating the orientation of the object (pallet) will be described later.

Next, how to obtain the coordinates of the 3D reference point will be explained using FIG.

This is the process corresponding to S20 in FIG.

First, in the process of obtaining the 3D reference point in S20, the intersection of straight lines passing through two mutually opposing vertices of A ₁₁ , A ₁₂ , A ₂₁ , and A ₂₂ recognized by 2D coordinates is taken, and this is referred to as the "apparent palette "front center point". In this embodiment, the apparent pallet front center point 14 is used as the 3D reference point.

Let the 2D coordinates of A ₁₁ , A ₁₂ , A ₂₁ and A ₂₂ be A ₁₁ =(x ₁₁ ,y ₁₁ ), A ₁₂ =(x ₁₂ ,y ₁₂ ), A ₂₁ =(x ₂₁ ,y ₂₁ ), If A ₂₂ =(x ₂₂ , y ₂₂ ), the 2D coordinates (x ₀ , y ₀ ) of the apparent pallet front center point 14 can be obtained by solving the following simultaneous equations (Equation 1).

Then, by using 3D software, the 3D coordinates (x ₀ , _y 0 , z ₀ ) of the apparent pallet front center point 14 corresponding to the 2D coordinates (x ₀ , y ₀ ) of the apparent pallet front center point 14 Ask for

This process can be obtained, for example, by using the rs2_deproject_pixel_to_point function provided by Intel's RealSense (registered trademark) technology. The rs2_deproject_pixel_to_point function receives as arguments the camera internal parameter object, the coordinates (X, Y) in the image, and the depth, and transforms them into a space with the camera as the origin.

Next, using FIGS. 6 and 7, the processing for calculating the posture of the object (pallet) from the 2D coordinates of one of the four vertices and the 3D reference point will be described.

This is the process corresponding to S30 in FIG.

As a coordinate system, the horizontal direction of the 3D camera 22 is taken as the X axis, the right side as the X + direction, the front and back direction as the Z axis, the front as the Z + direction, the vertical direction as the Y axis, and the bottom as the Y + direction. I will take it.

Here, it is assumed that the pallet 10 is rotated by _θy in the Y-axis direction.

The positional relationship between the 3D camera and the pallet as seen from behind the 3D camera is shown in FIG. Also, the positional relationship between the 3D camera and the palette as viewed from above the 3D camera is as shown in FIG.

At this time, assuming that the coordinates of the upper left vertex a ₂₁ of the palette 10 are (x _t , y _t , z _t ), there is a relationship shown in (Equation 2) below.

Here, h is the height of the pallet 10 and s is the width of the pallet 10, as described with reference to FIG.

Next, consider the relationship between the coordinates of the upper left vertex a ₂₁ of the palette 10 (x _t , y _t , z _t ) and the upper left vertex A ₂₁ =(x ₂₁ , y ₂₁ ) obtained by image recognition processing.

Between these two, there is the following relationship (Formula 3).

Here, r _v is the left and right resolution of the 2D image (e.g., 1920 pixels), r _h is the top and bottom resolution of the 2D image (e.g., 1080 pixels), α is the top and bottom angle of view of the 2D image (e.g., 59 degrees ), and β is the left and right angle of view (for example, 90 degrees) of the 2D image.

This formula describes the relationship between 2D coordinates and 3D coordinates based on the properties of 2D images.

Eliminate (x _t , y _t , z _t ) from (Equation 2) and (Equation 3), and use the following (Equation 4), which is a trigonometric function trivial relational expression, to obtain 0≦θ _y <π _θy , which is the rotation angle of the pallet 10 in the Y-axis direction, can be obtained.

The rotation of the pallet 10 in the three-dimensional axial direction also exists in the X-axis and Z-axis directions. Therefore, in order for the forklift 20 to recognize the position of the pallet 10, it is considered sufficient to consider only the rotation angle of the shaft in the vertical direction (the pallet moves left and right).

As described above, according to the present embodiment, by using a 2D image obtained from a 2D camera and 3D data obtained from a 3D camera, the orientation of a pallet on which packages are to be loaded can be determined with high accuracy and accuracy through simple calculations. Therefore, it is possible to smoothly carry out the transportation work using cargo handling equipment such as a forklift.

[Embodiment 2]
A second embodiment according to the present invention will be described below with reference to FIGS. 8 to 11. FIG.

In the first embodiment, a 2D image obtained from a 2D camera and 3D data obtained from a 3D camera are used to calculate the angle of the pallet from the coordinate system.

This embodiment corrects the 3D reference point used in the calculation of the first embodiment so that it can be calculated more accurately.

In this embodiment, the points different from the first embodiment will be mainly described.

The configuration of the forklift transportation system at the logistics site is almost the same as that of the first embodiment. However, a 3D reference point correction unit 106 is added to the object orientation calculation device 100 . The 3D reference point correction unit 106 is a functional unit that obtains a corrected pallet front surface center point from the apparent pallet front surface center point 14 of the first embodiment (details will be described later).

Next, the processing of the object orientation calculation device in Embodiment 2 will be described using FIGS. 9 to 11. FIG.

The object orientation calculation processing of the second embodiment is substantially the same as the object orientation calculation processing of the first embodiment shown in FIG. 4, but 3D reference point coordinate correction processing (S21) is added between S20 and S30. .

The 3D reference point coordinate correction process is a process of obtaining a corrected pallet front center point from the apparent pallet front center point 14 .

The 3D reference point coordinate correction process will be described in detail below with reference to FIGS. 10 and 11. FIG.

In the 3D reference point coordinate correction processing of this embodiment, as shown in the first embodiment, the intersection point of straight lines passing through the two opposing vertices of A ₁₁ , A ₁₂ , A ₂₁ , and A ₂₂ recognized by 2D coordinates It is assumed that the apparent center point 14 of the front face of the pallet is obtained.

Next, let L ₁ be the length of the line segment between A ₁₁ and the apparent center point 14 of the pallet front surface, and L ₂ be the length of the line segment between the apparent center point 14 of the pallet front surface and A ₂₂ .

L ₁ and L ₂ are obtained by the following (Equation 5).

Then, L ₁ /L ₂ is determined as a ratio of these line segments, and a corrected pallet front center point 15 is determined from the apparent pallet front center point 14 according to this line segment ratio L ₁ /L ₂ .

The coordinates of the corrected pallet front surface center point 15 are (x _a , _ya , za ₎ , and the correction amounts of the apparent pallet front surface center point 14 in three-dimensional coordinates are (Δ _x , Δ _y , Δ _z ). Then, the following (formula 6) is established.

In order to obtain the correction amounts (Δ _x , Δ _y , Δ _z ) from the line segment ratio L ₁ /L ₂ , for example, for each line segment ratio L ₁ /L ₂ , the center point correction data shown in FIG. Data such as a table should be retained and obtained.

As shown in FIG. 11, the center point correction data table 1131 defines an X correction amount _Δx 1131b, a Y correction amount _Δy 1131c, and a Z correction amount _Δz 1131d for each line segment ratio L ₁ /L ₂ 1131a. It is a table that has been

Appropriate values for the X correction amount _Δx 1131b, the Y correction amount _Δy 1131c, and the Z correction amount _Δz 1131d may be obtained by, for example, statistical data processing.
In the above description, an example of obtaining the correction amount from the data of the table in which the correction amount for each line segment ratio L ₁ /L ₂ is defined was explained, but with the line segment ratio L ₁ /L ₂ as an argument, A library function that returns the correction amount as a value may be prepared to obtain the correction amount for the line segment ratio L ₁ /L ₂ .

The processing of calculating the posture of the object (pallet) from the 2D coordinates of one of the four vertices seen from the forklift 20 side in S30 and the 3D reference point (corrected center point of the front face of the pallet) is , is the same as in the first embodiment.

However, instead of (Equation 2) in Embodiment 1, the following (Equation 7) in which the coordinates are replaced is used.

In this embodiment, in order to correct the apparent pallet front center point 14 obtained from the four vertices A ₁₁ , A ₁₂ , A ₂₁ , and A ₂₂ of the pallet and adopt a more accurate 3D reference point, as a result , a more accurate pallet orientation can be obtained.

[Embodiment 3]
A third embodiment according to the present invention will be described below with reference to FIG.
This embodiment, like the second embodiment, corrects the 3D reference points used in the calculation of the first embodiment to more accurately calculate them.

In this embodiment as well, the points different from the first embodiment will be mainly described.

The configuration of the forklift transportation system at the distribution site and the processing of the object orientation calculation device are the same as in the second embodiment.

In this embodiment, the contents of the 3D reference point coordinate correction process (S21) are different from those in the second embodiment.

The 3D reference point coordinate correction process will be described in detail below with reference to FIG.
In the 3D reference point coordinate correction processing of this embodiment as well, the apparent center point 14 of the pallet front surface, which is the intersection of straight lines passing through the two opposing vertices of A ₁₁ , A ₁₂ , A ₂₁ , and A ₂₂ recognized by 2D coordinates, is required.

Here, from the apparent center point 14 of the front face of the pallet, a point group cutting area 16 (that is, height 2t, width 2p) having a length t in the vertical direction and a length p in the horizontal direction is formed.

Then, if the image obtained in S02 of FIG. 3 has m 3D coordinate points in the point cloud cut region 16, each (X, Y, Z ) coordinates to obtain the coordinates (x _a , y _a , za ₎ of the corrected pallet front center point 15 .

where (x _i , y _i , z _i ) (i=1, .

Also, the point cloud cutting area 16 takes an area of an appropriate size when viewed from the shape of the palette. For example, the length t in the up-down direction and the length p in the left-right direction are given by the following (Equation 9).

Here, _Hh is the height of the fork pocket 11, _Hs is the width of the fork pocket 11, and _Rs is the width of the rectangular region between the fork pockets.

The processing of calculating the posture of the object (pallet) from the 2D coordinates of one of the four vertices seen from the forklift 20 side in S30 and the 3D reference point (corrected center point of the front face of the pallet) is , Embodiment 1, and Embodiment 2.

Also, instead of (Equation 2) of Embodiment 1, using (Equation 7) in which the coordinates are replaced is the same as in Embodiment 2.

In this embodiment, as in the second embodiment, the apparent pallet front center point 14 obtained from the four vertices A ₁₁ , A ₁₂ , A ₂₁ , and A ₂₂ of the pallet is corrected to obtain a more accurate 3D reference point. As a result, a more accurate pallet orientation can be obtained for adoption.

5 access point,
10... Pallet, 11... Fork pocket, 12... Pallet side, 13... Rectangular area between fork pockets, 14... Apparent pallet front center point, 15... Corrected pallet front center point, 16... Point group cutting area,
20... Forklift 21... 2D camera (RGB camera) 22... 3D camera (Depth camera) 25... Fork 27... External interface unit 28... Drive wheel 26... Fork operation unit 30... Operation command unit 40... radar sensor,
100... Object orientation calculation device,
101... Image vertex recognition unit 102... 3D reference point determination unit 103... Object angle calculation unit 104... Image reception unit 105... Object angle transmission unit 106... 3D reference point correction unit 110... Storage unit 111... Image data DB 112 Measurement data DB 113 Object orientation calculation data DB

Claims

1. An object attitude calculation device that inputs a two-dimensional image captured by a two-dimensional camera and a three-dimensional image captured by a three-dimensional camera and calculates an angle of a target object with respect to a predetermined coordinate system,
an image vertex recognition unit that recognizes a plurality of vertices of a target object from the two-dimensional image;
a three-dimensional reference point determining unit that determines three-dimensional coordinates of a three-dimensional reference point based on the coordinates of vertices recognized by the image vertex recognition unit;
an object orientation calculation unit that calculates an angle of one side surface of the target object with respect to a predetermined axis with respect to the predetermined coordinate system;
The object attitude calculation unit calculates the coordinates of vertices in three-dimensional coordinates, constraints on three-dimensional reference points in three-dimensional coordinates, and constraints on two-dimensional coordinates of vertices and corresponding three-dimensional coordinates based on the properties of the two-dimensional image. and calculating the angle based on a condition.
The object orientation calculation device according to claim 1, wherein the target object has a rectangular parallelepiped shape, and the vertices recognized by the image vertex recognition unit include four vertices on one side of the rectangular parallelepiped shape.
3. The object attitude calculation apparatus according to claim 2, wherein the three-dimensional reference points are arranged at intersections of line segments connecting two opposing points of the four vertices of the side surfaces.
With respect to the intersection point of the line segment connecting the two opposing points of the four vertices of the side surface, the length of the first line segment connecting the two opposing points and the length of the second line segment 3. The object attitude calculation apparatus according to claim 2, wherein the point corrected based on the ratio is used as the three-dimensional reference point.
Image data is acquired for a certain area centered on the intersection of line segments connecting two opposing points of the four vertices of the side surface, and the arithmetic mean of each coordinate in the three-dimensional coordinates of the image data. 3. The object attitude calculation apparatus according to claim 2, wherein the three-dimensional reference point is a point whose coordinates are the values obtained by taking the following values.
The object orientation calculation device according to claim 1, wherein the target object is a pallet.
The object orientation calculation device according to claim 5, wherein the target object is a pallet, and the fixed area includes a rectangular area between fork pockets of the pallet.
a two-dimensional camera that captures a two-dimensional image;
a three-dimensional camera that captures a three-dimensional image;
a forklift and
A forklift transportation system comprising an object attitude calculation device that inputs a two-dimensional image captured by a two-dimensional camera and a three-dimensional image captured by a three-dimensional camera and calculates the angle of the target object with respect to a predetermined coordinate system. and
a step in which the object orientation calculation device obtains a two-dimensional image captured by a two-dimensional camera;
obtaining a three-dimensional image captured by a three-dimensional camera;
an image vertex recognition step in which the object orientation calculation device recognizes a plurality of vertices of a palette from the two-dimensional image;
a three-dimensional reference point determination step in which the object orientation calculation device determines the three-dimensional coordinates of a three-dimensional reference point based on the coordinates of the vertices recognized by the image vertex recognition unit;
an object orientation calculation step in which the object orientation calculation device calculates an angle of one side surface of the target object with respect to a predetermined axis with respect to the predetermined coordinate system;
an object orientation calculation unit that calculates an angle of one side surface of the target object with respect to a predetermined axis with respect to the predetermined coordinate system;
a step in which the object attitude calculation device transmits the calculated angle to the forklift;
the forklift maneuvering the pallet based on the transmitted angle;
In the object attitude calculation step, the coordinates of the vertices in the 3D coordinates, the constraints of the 3D reference points in the 3D coordinates, and the 2D coordinates of the vertices based on the properties of the 2D image and the corresponding 3D coordinates. and calculating the angle based on a condition.