JP2003202216A - Method, device, system and program for three-dimensional image processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To deal with the difficulty in accurately estimating the surface attribute of a real object for three-dimensional image processing. <P>SOLUTION: The method and device for three-dimensional image processing derives the data which presents the normal direction of each part of the object's surface based on the image information obtained by photographing the object 10 that is projected by structure light from a projecting means 40 and estimates the surface attribute of the real object using the data which presents the normal direction. According to the measurement result of the real object's shape or the data related to the real object's shape input by users, either one among the pattern type of the projecting structure light, the pattern size and the projecting position of the structure light with respect to the real object is made varied. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional image processing method and a three-dimensional image processing apparatus for creating a three-dimensional image of a real object.

[0002]

2. Description of the Related Art Recently, three-dimensional images or three-dimensional images are
Not only for entertainment purposes such as movies and events, but also for business purposes in product design and urban development, as well as for consumer purposes, high presence such as virtual malls and 3D display product catalogs on the web, more accurate transmission of information The need is increasing for the purpose.

However, most of the current three-dimensional images are three-dimensional computer graphics that are manually created from the beginning without taking in the actual shape data and surface attribute data of an object. It takes a lot of time and cost to create such three-dimensional computer graphics, which hinders the spread of the three-dimensional computer graphics in the world.

Further, since three-dimensional computer graphics are modeled by the hand of the maker, when a three-dimensional object is made into a three-dimensional object, its shape and surface color are not accurately reproduced.

For this reason, a method of making a three-dimensional image model of a real object by actually measuring the shape of the real object, photographing a two-dimensional image at a position corresponding to the shape, and pasting the two-dimensional image on the three-dimensional shape data is popular. Is being researched.

In order to actually measure the shape and surface attributes of an actual object, it is necessary to measure the three-dimensional shape and acquire surface texture information.

As a main three-dimensional measurement method, a slit light projection method or a pattern projection method is known.
These methods are one type of active measurement methods in which a specific reference light (also referred to as detection light) is applied to an object to be measured and three-dimensional coordinates are obtained by the principle of triangulation.

The slit light projection method scans an object to be measured by irradiating and deflecting slit light.
Moreover, in the pattern projection method, a plurality of two-dimensional pattern lights are sequentially irradiated. Then, the three-dimensional coordinates are calculated by analyzing the image of the reflection pattern on the surface of the object in the images obtained by time-sequentially scanning the slit light scan and the images obtained by sequentially irradiating the pattern light.

On the other hand, as a method for obtaining the texture information of the surface of the object, a photographing device having an optical system for picking up a color image of the object, an image sensor, etc. is used. Then, a three-dimensional image model is constructed by fusing the above three-dimensional coordinate data and the texture information acquired by the photographing device.

However, much research has not been done on a method of acquiring even surface attribute information such as the texture and glossiness of an actual object. Conventionally, the surface attribute is simplified into an object color to express the color of a two-dimensional light source. The system is used as it is. In this method, due to the influence of the position of the illumination projection source and the spectral characteristics at the time of capturing the two-dimensional image, the tint changes, and specular reflection or illuminance distribution causes an error in luminance.

In view of these problems, a method for creating a three-dimensional image model including surface attributes has been proposed. According to this, not only the shape, texture, and color of the target real object, but also the observation environment at the time of reproduction, that is, the shape, position, direction, and color of the illumination light, and the position and direction of the observer are considered. Surface attributes can be measured.

In this three-dimensional image model creating method, the surface reflection is modeled so as to be represented by two components, a surface reflection component and an internal (diffuse) reflection component, and the object reflection when the illumination condition is changed is imaged. The surface attribute of the object is estimated by analyzing the image.

Here, the surface reflection component has a strong reflection intensity in the regular reflection direction of the illumination light, and the internal reflection component has a property of being isotropically reflected to the outside after the light penetrates into the object.
As described above, the surface attribute of the object is sensitive to the angle formed by the normal line direction of the object surface. Therefore, in order to correctly measure the surface attributes of the target object, each part (each position) of the target object surface must be measured.
It is necessary to correctly measure the normal direction at.

[0014]

However, in the above-described three-dimensional image model creating method, a method for accurately acquiring the normal direction of the surface of the target object has not been realized,
There is a possibility that a large error may be included in the measurement of the surface attribute.

Although it is possible to obtain the normal direction by using the three-dimensional shape data, the process of calculating the three-dimensional normal vector from the shape data is extremely complicated.
It has the disadvantage that it takes time to calculate.

Further, the three-dimensional shape data is generally displayed by a polygon model, but if the normal direction is represented by a polygon surface, the spatial distribution of surface attributes depends on the polygon shape. The problem arises.

Therefore, the present invention provides a three-dimensional image processing method and a three-dimensional image processing apparatus capable of reproducing a more realistic surface attribute of an actual object by measuring the normal direction of the surface of the actual object with high accuracy. It is intended to be provided.

[0018]

In order to achieve the above object, in the three-dimensional image processing method and the three-dimensional image processing apparatus of the present invention, a real object onto which structured light from the projection means is projected is photographed. Based on the obtained image information, data indicating the normal direction of each surface portion of the real object is acquired, and the surface attribute of the real object is estimated using the data indicating the normal direction.

In this way, a line pattern is formed on the real object,
It is possible to accurately determine the normal direction of each part of the actual object by performing an analysis using the structured light reflection pattern included in the image information captured by projecting the structured light having a texture pattern and a grayscale pattern. Becomes

Specifically, for example, the reflected light pattern of the projected structured light by the real object is photographed, and the normal direction on the surface of the real object is calculated based on the distortion of the reflected light pattern in the image. What kind of structured light is to be projected, as long as it is possible to calculate the inclination (that is, the normal direction) of the surface of the real object from the distortion of the reflected light pattern by projecting it on the real object like a square lattice pattern? Structured light may be used.

By thus accurately determining the normal direction of each part of the real object, the real object estimated based on the positional relationship among the real object, the projection means, and the photographing device based on the normal direction is used. It is possible to make surface attributes such as color, gloss, and brightness highly accurate, that is, more realistic and closer to an actual object.

It should be noted that the three-dimensional shape data of the real object may be obtained based on the measurement result of a dedicated device for measuring the shape of the real object using the optical radar method, the interferometry method, etc.
Based on the image information including the reflection pattern of the structured light,
If not only the surface attribute data but also the 3D shape data of the real object is generated, it is not necessary to separately acquire the 3D shape data and the image capturing for acquiring the surface normal direction. It is possible to create a three-dimensional image with.

Further, according to the measurement result of the shape of the real object or the data on the shape of the real object input by the user, the type of the pattern of the structured light to be projected, the size of the pattern, and the projection of the structured light onto the real object. By changing one of the positions, the projection state of the structured light is optimized according to the shape (complexity, etc.) of the real object, and the normal direction of each part of the real object surface can be obtained with higher accuracy. It should be set to.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION (First Embodiment) FIG. 1 shows the configuration of a three-dimensional image processing system according to the first embodiment of the present invention. In this three-dimensional image processing system, an object (real object) 10 is placed, and a rotatable stage 2 is rotatable.
0, a slit light source 32 for irradiating the object 10 with slit light and a movable slit 34, and a slit light projector 30 capable of controlling these, and the object 10
A structured light projector 40 for projecting structured light onto the illuminating device, an illuminating light source 52, and a lighting device 50 capable of turning on / off the light source and controlling the position of the illuminating light source 52;
The image capturing device 60 includes a CD, a CMOS, and the like, and includes a digital camera that captures an image of the object 10, and a three-dimensional image processing device 200 including a computer and the like.

The three-dimensional image processing apparatus 200 has a user interface (operating means) 7 that can be operated by an operator.
0, a controller 80 that controls the rotary stage 20, the slit light projector 30, the structured light projector 40, and the illumination device 50, an image memory 90 that stores image information captured by the image capturing device 60, and an image memory 90. A three-dimensional shape calculation unit (three-dimensional shape acquisition unit) 100 that calculates three-dimensional shape data of the object 10 based on the stored image information.
It has and.

Furthermore, the three-dimensional image processing apparatus 200 uses the object 1 based on the image information stored in the image memory 90.
A normal direction calculation unit (normal direction acquisition unit) 110 that calculates a normal direction of each surface position of 0, and a normal direction calculation unit 110.
A surface attribute calculation unit (surface attribute acquisition unit) 120 that estimates and calculates the surface attribute of the object 10 based on the normal direction calculated by the above and the image information stored in the image memory 90.
A three-dimensional model data construction unit 130 that constructs three-dimensional image (three-dimensional model) data by integrating the three-dimensional shape data calculated by the three-dimensional shape calculation unit 100 and the surface attribute estimated by the surface attribute calculation unit 120; The output terminal 140 outputs the three-dimensional model data constructed by the three-dimensional model data constructing unit 130 to a display or a printing device (not shown).

Here, the strict positional relationship among the slit light source 32, the movable slit 34, the structured light projector 40, the illumination light source 52, and the image pickup device 60 is known.

Rotating stage 20, slit light projector 3
0, structured light projector 40, illumination device 50, and imaging device 6
0 is installed in a dark room or in a housing that blocks intrusion of light from the outside. The three-dimensional image processing device 200 is
It may be installed in the dark room or the housing, or may be installed outside the dark room or the housing.

Next, the creation of the three-dimensional model data of the object 10 by the three-dimensional image processing system configured as described above will be described with reference to the flow charts of FIGS. 2A shows a main processing flow, FIG. 2B shows a pre-shooting (step 2) subroutine, and FIG. 2 (C) shows a main shooting (step 4) subroutine. Further, the solid line in FIG. 1 indicates the processing flow of the above flow chart, and the double line indicates the data flow.

The object 10 is installed on the rotary stage 20. User interface by operator 7
When a start command for creating three-dimensional model data is issued by operating 0, a CPU (not shown) in the controller 80
Causes the photographing device 60 to start photographing.

Here, first, pre-imaging for measuring the complexity of the surface shape of the object 10 is performed. When the pre-photographing is started, the CPU causes the slit light projector 30 to turn on the slit light source 32, and after the driving power source of the slit light source 32 becomes stable, the slit light scanning on the object 10 by the movable slit 34 is started (step). <Abbreviated as S in the figure> 21). Then, in synchronization with the scanning of the slit light, the photographing device 60 is caused to photograph the reflection pattern image of the slit light of the object 10.

At this time, the slit light scans the surface of the object 10 one-dimensionally at a constant speed, and during the scanning, the photographing device 60.
Acquires an image of the object 10 at a constant frame rate (step 22).

In the pre-photographing, the scanning speed of the slit light or the image-capturing frame rate of the image-capturing device 60 is set with emphasis on reduction of processing and speed-up.

The slit light source 32 may be any light source such as a white light source or a laser light source as long as the slit light reflection pattern is observed at a sufficiently high S / N ratio during image capturing by the image capturing device 60. You can use it. Also,
Although the timing of lighting is arbitrary, it may be always lighting in order to eliminate a waiting time until the driving power source of the light source is stabilized.

When one photographing, that is, photographing of the object 10 from one observation direction is completed, the rotary stage 20
Until the rotation of 360 ° is completed (step 23), the rotary stage 20 is rotated by a predetermined angle (step 24),
The photographing device 60 is caused to photograph the reflection pattern image of the object 10 again. The captured images are sequentially stored in the image memory 90.

Then, when this process is repeated and the rotary stage 20 is rotated by 360 °, the three-dimensional shape calculation unit 110 obtains a plurality of reflection pattern images stored in the image memory 90.
3D coordinate data (3D shape data) of the surface of the object 10 by processing and analyzing it based on the triangulation principle.
To get

The three-dimensional coordinate data is sent to the three-dimensional shape complexity determination unit 100, and the three-dimensional shape complexity determination unit 100 analyzes the three-dimensional coordinate data and determines the complexity of the shape (complexity). Then, the optimum values of the scanning speed of the slit light and the stage rotation pitch in the subsequent actual photographing are estimated (step 3) and transferred to the controller 80.

In estimating the complexity of the shape in the three-dimensional shape complexity determining unit 100, the reflection pattern image is directly obtained from the image memory 90 without using the three-dimensional coordinate data obtained by the three-dimensional shape calculating unit 110. May be obtained, the amount of change between frames may be calculated to determine the continuity of the reflection pattern, and the complexity of the three-dimensional shape may be determined. In addition, any method may be adopted as long as the complexity of the three-dimensional shape is effectively determined.

In step 3, instead of calculating the slit light scanning speed, the frame rate of the photographing device 60 may be changed and set to an optimum value.

Then, the main photographing of the object 10 is started (step 4). The shooting here is a shooting of a slit light projection image for acquiring a more detailed three-dimensional shape than the pre-shooting of the object 10, and a shooting of a structured light projection image for acquiring the normal direction of the surface of the object 10. Then, the images used for estimating the surface attributes of the surface of the object 10 are captured in the order named (steps 5 and 6).

Further, each photographing is sequentially performed for each rotation (steps 7 and 8) of the stage rotation pitch calculated in step 3 of the rotary stage 20.
After one round (step 7), the main shooting ends.

When the controller 80 issues a command to start the main photographing, the slit light projector 30 starts scanning the object 10 with slit light as shown in FIG. 2B (step 41). Further, in synchronization with the scanning of the slit light, the photographing device 60 starts photographing the slit light reflection pattern image of the object 10.

The photographing specifications at this time are the same as those of the slit light reflection pattern photographing performed in the pre-photographing. However,
The scanning speed of the slit light or the imaging frame rate is the value determined in step 3.

The obtained slit light reflection pattern image is stored in the image memory 90, and the image data is transferred to the three-dimensional shape calculation unit 110. Three-dimensional shape calculation unit 11
0 calculates three-dimensional coordinates based on the triangulation principle (step 5). At this time, as the three-dimensional shape data,
For example, the specification may be such that processing is performed so that it can be expressed as a surface model using polygons or a set of surface shape elements having different shapes.

Further, in steps 43 and 44, photographing is performed under the structured light projection by the structured light projector 40. At this time, the light from the slit light source 32 is completely blocked or turned off with respect to the object 10. Structured light projector 40
Projects a square lattice structured light onto the object 10. A structured light reflection pattern image of the object 10 is picked up by the image capturing device 60, and the structured light reflection pattern image is stored in the image memory 90. Then, the structured light reflection pattern image is transferred to the normal direction calculation unit 120, and the normal direction calculation unit 120 calculates each intersection (lattice point) of the square lattice structured light on the surface of the object 10 based on the structured light reflection pattern image. ), The normal direction at the projection point is calculated (step 5).

When the normal direction calculation unit 120 calculates the normal direction, the position of the photographing device 60, the position of the structured light projector 40, the optical axis direction of the optical system of the photographing device 60, and the structured light projection. The angle formed by the structured light projection direction of the machine 40, the position and rotation angle data of the rotary stage 20, and the captured structured light reflection pattern image data are used.

If projective geometry is also used, when the surface of the object 10 is a plane, the coordinate of the photographed structured light reflection pattern image and the position and angle data of each of the above-mentioned devices are used to determine the method of that plane. Line direction data can be obtained.

Even in the case of the object 10 having a non-planar surface, the normal vector can be obtained by the following processing.

In FIG. 3, the structured light projector 40 is attached to the object 10.
The example of the structured light reflection pattern image imaged by the imaging device 60 at the time of projecting the square lattice image by is shown. The structured light reflection pattern image reflects the surface shape (curved surface shape) of the object and is distorted as a whole.

However, since it can be regarded as a plane in a minute area, the periphery of the projection point P of the grid point is regarded as a plane, and the normal line is calculated using the projection point P and the coordinate data of the nearest projection points Q _{1 to} Q _4. It is possible to find the direction.

A simple example of calculating the normal vector will be specifically described. An object surface unit normal vector at the lattice point P and vectors connecting the points Q ₁ Q ₄ and Q ₂ Q ₃ are n and Q, respectively.
_{When 1} Q ₄ and Q ₂ Q ₃ are set, it is possible to obtain the normal direction by solving | n | ² = 1 n · Q ₁ Q ₄ = 0 n · Q ₂ Q ₃ = 0.
Although the normal vector n obtained heavy solution (positive or negative coordinate component is highlighted), the internal facing or external direction of the vector n is the object by referring to the coordinate data of the photographing direction and the point Q ₁ to Q ₄ of the object Since it can be determined whether or not the normal direction can be determined.

By performing such normal vector calculation at the projection points of each grid point, the normal vector of each part of the surface of the object 10 is calculated. Further, in this method, since the normal direction can be obtained only at the projection points of each grid point, the projection points of the grid points can be shifted on the surface of the object 10 in order to find the normal direction at more points. As described above, the structured light projector 40 is provided with a function of slightly shifting the projection light, or a zoom optical system is mounted so that the magnification of the projection light can be changed so that the size of the projection pattern can be changed. Good.

In addition to the above method, any method may be used as long as it is a process capable of calculating the normal direction of the surface of the object 10.

Next, in steps 45 and 46, the lighting device 5
0 is driven to perform photographing under illumination light irradiation by the illumination light source 52. At this time, the light from the structured light projector 40 is the object 1
0 is completely shut off or turned off.

The illumination device 50 also sets the illumination light source 52 to the initial position to illuminate the object 10, the image of the object 10 is picked up by the photographing device 60, and the image is stored in the image memory 90.

The illuminating device 50 has a step angle determined in sequence,
The illumination light source 52 is rotated about the position of the object 10 in a direction different from the rotation direction of the rotary stage 20 (step 4).
7, 48). Then, similarly, the object 10 is irradiated with illumination light.
Image is captured. This process is repeated until the position of the illumination light source 52 reaches a predetermined maximum position (step 4).
7). Further, the position of the illumination light source 52 is transferred to the surface attribute calculation unit 130 together with the corresponding image data.

The surface attribute calculator 130 uses the Phong model and
Based on the dichroic reflection model such as the Torrance-Sparrow model, the photographing is performed with the normal direction at the starting point (projection point of the grid point) of each normal vector calculated in step 5 on the surface of the object 10 as a reference. The surface attribute at the starting point is estimated using the positional relationship between the target object 10, the illumination light source 52, and the imaging device 60 at that time.

The image photographed under irradiation of illumination light is the position data of the illumination light source 52 at this time, and the normal direction calculation unit 120.
With the normal direction data of the object 10 calculated by
In the surface attribute calculation unit 130, the color / gloss of the object 10
It is used for estimation calculation of surface attributes such as brightness.

In the three-dimensional image processing system of this embodiment, the shape of the object 10 is measured by the slit light projection image, the normal direction of the surface of the object 10 is calculated by the structured light projection image, and the illumination light irradiation image is used. Since the surface attribute estimation of the surface of the target object 10 is performed independently, the rotation stage 20 is rotated by a predetermined rotation angle and the imaging of the illumination light irradiation image is performed at each angular position in the same manner as described above.

The rotation angle of the rotary stage 20 is 360
The actual shooting ends at the point of time (step 7). Although the value of 360 degrees is set here, if the required three-dimensional model data is only in a certain range of the object 10, the main stage photography may be terminated when the rotary stage 20 rotates by that angle. Good.

The three-dimensional shape data calculated by the three-dimensional shape calculating section 110 in this way and the surface attribute calculating section 130
Each azimuth surface attribute data calculated by the above is transferred to the three-dimensional model data construction unit 140.

The three-dimensional model data constructing unit 140 integrates and optimizes the three-dimensional shape data and the surface attribute data obtained at each photographing direction (stage rotation position) (step 9) to optimize the three-dimensional model of the object 10. Build the data (step 10).

The acquired three-dimensional model data is stored in the memory in the three-dimensional model data construction unit 140, and is transferred to the output terminal 150 when a processing command is generated by the operator operating the user interface 70 (step 11). ).

As described above, according to the present embodiment, the structured light is projected onto the object 10 and the captured image of the reflected light pattern is analyzed to determine the normal direction of each part of the surface of the object 10. Can be obtained accurately. Therefore, the surface attribute of the object 10 can be estimated with high accuracy based on the normal direction data.

Further, in this embodiment, since the three-dimensional shape data of the object 10 and the normal direction of the surface of the object 10 are independently acquired, the resolution of the acquisition of the three-dimensional shape data, Without being affected by accuracy and specifications,
The surface attribute of the object 10 can be acquired with desired accuracy.

The three-dimensional image processing system of the present invention is not limited to the one described in this embodiment, and for example, the normal direction of the surface of the object can be obtained. In such a case, structured light other than the square lattice shape may be used, such as a circular texture pattern or other texture patterns arranged periodically, or a light and shade pattern such as a sine wave chart.

In addition, the laser radar method may be used instead of the slit light projection method to obtain the three-dimensional shape of the object.
It is also possible to use a multi-view stereoscopic method using a plurality of imaging devices.

Further, any structure and specification may be adopted as long as the present invention can be implemented.

(Second Embodiment) FIG. 4 shows a second embodiment of the present invention.
1 illustrates a configuration of a three-dimensional image processing system that is an embodiment. In the present embodiment, constituent elements common to those of the first embodiment are designated by the same reference numerals as those of the first embodiment, and will not be described.

The three-dimensional processing apparatus 200 of the first embodiment described above.
Then, using the captured image under the slit light projection, the object 1
The three-dimensional shape of 0 was measured, and the normal direction of the surface of the object 10 was measured using the captured image under the structured light projection. However, in the three-dimensional image processing apparatus 200 ′ of the present embodiment, the structured light projection is performed. Both the three-dimensional shape and the normal direction are measured using the photographed image below.

In the first embodiment, the case where the three-dimensional shape complexity determination unit 100 is provided and the complexity of the shape of the object 10 is determined from the measurement result of the three-dimensional shape of the object 10 in the pre-imaging will be described. However, in the present embodiment, the operator inputs the complexity of the shape of the object 10 via the user interface 70. Therefore, the three-dimensional shape complexity determination unit 100 is not provided. Then, in the present embodiment, the size of the structured light pattern projected on the object 10 is changed by changing the zoom magnification of the optical system in the structured light projector 40 according to the input complexity. There is.

The structured light projector 40, as shown in FIG.
Projection light source 42, square lattice slit 44, projection light source 4
Collimator lens 46 for converting the light from 2 to parallel light
And a zoom lens 48 that projects the projection light at an arbitrary magnification ratio.
It has and.

Further, the structured light projector 40 has a function of controlling the square lattice slit 44 at an arbitrary position in the plane orthogonal to the parallel rays passing through the slit 44.

The position control of the square lattice slit 40 and the zoom control of the zoom lens 48 are performed by a signal from the controller 80 shown in FIG.

The optical system in the photographing device 60 also has a zoom function, and the controller 80 can control the zoom magnification according to the zoom ratio of the structured light projector 40. The data such as the principal point position and focal length in all zoom ratios of the optical system are stored in a memory (not shown) in the controller 80.

When the object 10 is placed on the rotary stage 20 and imaging is started, the operator inputs the complexity (or data resolution) of the object shape. As a result, the CPU (not shown) in the controller 80 calculates the optimum zoom magnification of the optical system of the structured light projector 40 and the photographing device 60, and sets the optical system of the structured light projector 40 and the photographing device 60 to a predetermined value. Set the zoom magnification so that it can be obtained.

For example, if the complexity of the object 10 is high,
In order to reduce the zoom magnification (projection light enlargement ratio) of the structured light projector 40 to make the square grid pattern projected on the surface of the object 10 fine and to accurately shoot the reflected light pattern of the fine square grid, shooting is performed. Increase the zoom factor of device 60.

Next, with reference to the flowcharts shown in FIGS. 5A and 5B, the creation of three-dimensional model data by the three-dimensional image processing system of this embodiment will be described.
FIG. 5A shows a main processing flow, and FIG. 5B shows a main photographing (step 104) subroutine.

When the photographing start is instructed through the user interface 70 (step 101) and the operator inputs the complexity of the object shape (step 10).
2) As described above, the zoom magnification of the optical system of the structured light projector 40 and the imaging device 60 is set (step 1).
03).

Next, actual photographing is performed (step 104).
In the main photographing, as shown in FIG. 5B, first, a scanning projection of a square lattice image is performed on the object 10 by the structured light projector 40 (step 143). The scanning of the square lattice image is performed by one pitch of the lattice by driving the square lattice slit 44 shown in FIG. 6 vertically and horizontally. Further, the scanning is performed at a constant speed. Then, the object 10 under the structured light projection is imaged by the imaging device 60 at a constant frame rate, and the captured image is transferred to the image memory 90 (step 144). The image data is transferred from the image memory 90 to the three-dimensional shape calculation unit 110 ′ and the normal direction calculation unit 120.

The position and size of the square lattice slit 44, the position (zoom magnification) of the zoom lens 48 of the structured light projector 40, the principal point position of the optical system of the photographing device 60, the zoom magnification,
Since the position of the photographing surface and the like are known, if the reflected light from the object 10 of each lattice pattern in the direction perpendicular to the slit scanning direction is analyzed in the photographed image, the position of each surface of the object 10 on the basis of the triangulation principle is analyzed. The coordinates are obtained.

Further, the normal direction data of the surface of the object 10 can be obtained by analyzing the photographed image by the same method as in the first embodiment.

The structured light scanning is performed for one pitch of the grid or more, a large number of captured images are acquired, and the data to be analyzed is increased, so that the three-dimensional coordinates and the normal line of the surface of the object 10 can be more accurately measured. Direction data may be acquired.

Further, when the object 10 has a high degree of complexity and the structured light is projected in a reduced size in order to measure a complicated shape as accurately as possible, the projection pattern may cover the entire surface area of the object 10. When it is not possible, the structured light projector 40 and the photographing device 60 are position-controlled so as to acquire the structured light projection image of the entire area of the object 10 while keeping their relative positions. Good.

In this way, the three-dimensional shape calculation unit 11
In 0 ′, the three-dimensional shape data of the object 10 is calculated, and in the normal direction calculation unit 120, the normal direction data of the surface of the object 10 is calculated.

Subsequently, similarly to the first embodiment, the image of the object 10 is taken under the environment of the positions of the respective illumination light sources 52 and the respective rotational positions of the rotary stage 20 (step 14).
5, 146, 147, 148), the surface attribute estimation calculation based on the image data and the normal direction data of the surface of the object 10 (steps 106, 1), and further the integration of three-dimensional data in all directions. The surface attribute data is integrated, the three-dimensional model data is constructed, and the three-dimensional model data is finally output from the output terminal 150.

As described above, according to the present embodiment, the structured light is projected on the object 10 and the photographed image of the reflected light pattern is analyzed, whereby the three-dimensional shape data of the object 10 and the surface of the object 10 are analyzed. Both normal direction data can be acquired accurately and at high speed. Therefore, compared to a case where the three-dimensional shape data is measured by the three-dimensional measuring device using the laser radar method or the interferometric method and the normal direction data is acquired using the image captured by the image capturing device 60, The number of constituent devices can be reduced and the space can be saved, and a high-performance three-dimensional image processing system can be realized at low cost.

Further, as in the present embodiment, the structured light projector 4 is used.
0 and the zoom magnification of the optical system of the image capturing device 60 can be changed, so that even for an object having a complicated surface shape, its three-dimensional shape data and surface normal direction data can be accurately acquired. It is possible to realize a three-dimensional image processing system capable of highly accurately estimating the surface attribute of an object having a complicated surface shape.

The three-dimensional image processing system of the present invention is not limited to the one described in this embodiment, and any structure and specification can be adopted as long as the present invention can be implemented. Good.

[0090]

As described above, according to the present invention,
The normal direction of each part of the real object can be accurately obtained by using the reflection pattern of the structured light included in the image information obtained by projecting the structured light on the real object. Therefore, the surface attributes such as color, gloss, and brightness of the real object can be accurately determined based on the positional relationship between the real object, the structured light projection means, and the image capturing device, which are based on the normal direction thus accurately determined. That is, it is possible to estimate a realistic surface attribute that is closer to that of a real object.

If not only the surface attribute data but also the three-dimensional shape data of the real object is generated based on the image information including the reflection pattern of the structured light, the acquisition of the three-dimensional shape data and the surface There is no need to separately perform image capturing for acquiring the normal direction or to install a dedicated measuring device for acquiring three-dimensional shape data, and create three-dimensional images at high speed and at low cost. I can do it.

Further, according to the measurement result of the shape of the real object or the data on the shape of the real object inputted by the user, the type of the pattern of the structured light to be projected, the size of the pattern and the projection of the structured light onto the real object. If either of the positions is changed and the projection state of the structured light can be optimized according to the shape (complexity, etc.) of the real object, the normal line of each part of the real object surface will be more highly accurate. The direction and thus the surface attribute can be obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a three-dimensional image processing system which is a first embodiment of the present invention.

FIG. 2 is a flowchart showing a process of creating three-dimensional model data in the three-dimensional image processing system.

FIG. 3 is a schematic diagram showing a reflection pattern image captured when a square lattice image is projected on an object by the structured light projector of the three-dimensional image processing system.

FIG. 4 is a block diagram showing a configuration of a three-dimensional image processing system which is a second embodiment of the present invention.

FIG. 5 is a flowchart showing a process of creating three-dimensional model data in the three-dimensional image processing system according to the second embodiment.

FIG. 6 is a schematic diagram showing a configuration of a structured light projector that constitutes the three-dimensional image processing system of the second embodiment.

[Explanation of symbols]

10 objects 20 rotation stages 30 slit light projector 32 slit light source 34 Movable slit 40 Structured light projector 42 Projection light source 44 square lattice slit 46 Collimator lens 48 zoom lens 50 Lighting equipment 52 Illumination light source 60 Imaging device 70 User Interface 80 controller 90 image memory 100 three-dimensional shape complexity determination unit 110, 110 'three-dimensional shape calculation unit 120 Normal direction calculation unit 130 Surface Attribute Calculation Unit 140 3D model data construction unit 150 output terminals 200,200 'three-dimensional image processing apparatus

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 2F065 AA35 AA53 BB05 FF07 FF26                       HH05 HH06 LL06 LL28 PP13                       PP23                 5B057 BA02 BA15 BA17 CA13 CA16                       CB13 CB16                 5B080 AA00 AA13 DA06 GA15

Claims (19)

[Claims] 1. A first step of projecting structured light onto a real object to photograph the real object, and a normal direction of each surface portion of the real object based on image information photographed in the first step. A third order characterized by including a second step of acquiring the data shown, and a third step of estimating the surface attribute of the real object using the data showing the normal direction acquired in the second step. Original image processing method. 2. The first step in the second step
The three-dimensional image processing method according to claim 1, wherein the data indicating the normal direction is acquired based on the reflected light pattern of the structured light of the real object obtained from the image information captured in the step. 3. The three-dimensional image processing method according to claim 1, wherein the structured light has any one of a line pattern, a texture pattern, and a grayscale pattern. 4. A fourth step of generating three-dimensional shape data of the real object based on a reflected light pattern of the real object by the structured light, which is obtained from the image information photographed in the first step. The three-dimensional image processing method according to claim 1. 5. In the second step, any one of the type of the structured light pattern, the size of the pattern, and the position of the projection source is changed based on the measurement result of the shape of the real object. The three-dimensional image processing method according to claim 1. 6. In the second step, according to data indicating a surface shape of the real object input by an operator,
The three-dimensional image processing method according to claim 1, wherein any one of a type of the structured light pattern, a pattern size, and a projection position of the structured light with respect to the real object is changed. 7. A normal direction acquisition means for acquiring data indicating a normal direction of each surface portion of the real object based on image information obtained by photographing the real object onto which the structured light from the projection means is projected. And a surface attribute estimating means for estimating the surface attribute of the real object using the data indicating the normal direction obtained by the normal direction obtaining means. 8. The normal direction acquisition means acquires data indicating a normal direction based on a pattern of reflected light of the structured light of the real object obtained from the image information. The three-dimensional image processing device according to item 7. 9. The three-dimensional image processing apparatus according to claim 7, wherein the structured light has any one of a line pattern, a texture pattern, and a grayscale pattern. 10. The three-dimensional shape acquisition means for generating three-dimensional shape data of the real object based on a reflected light pattern of the real object, which is obtained from the image information, by the structured light. The three-dimensional image processing device according to item 7. 11. The normal direction acquisition means selects one of a pattern type of the structured light, a size of the pattern, and a projection position of the structured light with respect to the real object, according to a measurement result of the shape of the real object. The three-dimensional image processing apparatus according to claim 7, wherein any one of them is changed. 12. The structured light pattern has an operating means for allowing a user to input data relating to the surface shape of the real object, and the normal direction acquisition means has a pattern of the structured light according to the data input from the operating means. The three-dimensional image processing apparatus according to claim 7, wherein any one of the type, the size of the pattern, and the position of the projection unit is changed. 13. A projection means for projecting structured light onto a real object, a photographing device for photographing the real object and outputting image information, and the three-dimensional image processing device according to claim 7. A three-dimensional image processing system comprising: 14. A three-dimensional image processing program for operating a computer, comprising: a first step of projecting structured light onto a real object to photograph the real object; and image information photographed in the first step. A second step of acquiring data indicating the normal direction of each surface portion of the real object based on the above, and estimating the surface attribute of the real object using the data indicating the normal direction acquired in the second step A three-dimensional image processing program comprising a third step. 15. In the second step, acquiring data indicating a normal direction based on a reflected light pattern of the structured light of the real object obtained from the image information captured in the first step. 15. The method according to claim 14,
The three-dimensional image processing program described in. 16. The three-dimensional image processing program according to claim 14, wherein the structured light has any one of a line pattern, a texture pattern, and a grayscale pattern. 17. A fourth step of generating three-dimensional shape data of the real object based on a reflected light pattern of the real light of the real object, which is obtained from the image information photographed in the first step. Claim 1 characterized by the above-mentioned.
The three-dimensional image processing program according to item 4. 18. In the second step, any one of the type of the structured light pattern, the size of the pattern, and the position of the projection source is changed based on the measurement result of the shape of the real object. The three-dimensional image processing program according to claim 14. 19. In the second step, the type of pattern of the structured light, the size of the pattern, and the structured light with respect to the real object are input according to data indicating a surface shape of the real object input by an operator. 15. The three-dimensional image processing program according to claim 14, wherein any one of the projection positions of is changed.