JP6704700B2 - Image processing apparatus, control method thereof, and program - Google Patents

Description

The present invention relates to an image processing device, a control method therefor, and a program, and more particularly to a conversion technique for three-dimensional data.

Conventionally, a method is known in which the distance to each pixel area of a captured image is measured and three-dimensional data is acquired. For example, Patent Document 1 discloses a method for effectively acquiring three-dimensional data by combining distance measurement by a laser and a captured image.

JP, 2004-037396, A

By the way, in recent years, a technique called 3D printing, which can output the three-dimensional shape of the object as a modeled object, is becoming popular. If the output of 3D shaped objects becomes easy and spreads in households, it is possible to meet the needs of a particular type of 3D shape data by simply changing the scale or scale of the original data. It is possible that there is no output object.

The present invention has been made in view of the above-mentioned problems of the conventional art, and an image processing apparatus capable of transforming three-dimensional shape data for outputting a modeled object at a scale or a double scale suitable for the modeled object. It is intended to provide a control method and a program therefor.

In order to solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, an acquisition unit that acquires three-dimensional data representing a space including an object, a conversion unit that converts the position data of the three-dimensional data in a predetermined direction, and a conversion unit that converts the position according to the position where the object exists, and the predetermined direction. possess an output means for the position of the outputs of the three-dimensional data transformed, wherein the conversion means, if multiple objects are present, the data of the position of the range corresponding to spaces without the plurality of objects It is characterized in that the conversion is performed with a larger compression rate than the data of the position in the range corresponding to the space in which the plurality of objects exist .

According to the present invention, it is possible to transform three-dimensional shape data for outputting a modeled object at a scale or a double scale suitable for the modeled object.

The figure explaining the subject which should apply the conversion process of the three-dimensional data which concerns on this invention. FIG. 3 is a block diagram showing a functional configuration example of a digital camera 200 as an example of the image processing apparatus according to the first embodiment. 3 is a flowchart showing a series of operations of conversion processing of three-dimensional data according to the first embodiment. FIG. 6 is a diagram illustrating a detection process of a range of a main object according to the first embodiment. FIG. 3 is a diagram illustrating compression in the depth direction of three-dimensional data according to the first embodiment. Example of data format of three-dimensional data according to the first embodiment FIG. 3 is a diagram illustrating compression in the depth direction of three-dimensional data using polygon information according to the first embodiment. FIG. 3 is a diagram illustrating compression of three-dimensional data in a depth direction as a modified example according to the first embodiment. 3 is a flowchart showing a series of operations of conversion processing of three-dimensional data according to the second embodiment. FIG. 6 is a diagram illustrating a detection process of a range of a main object and a sub-main object in the second embodiment FIG. 5 is a diagram illustrating compression of three-dimensional data in the depth direction according to the second embodiment. 3 is a flowchart showing a series of operations of conversion processing of three-dimensional data according to the third embodiment. FIG. 8 is a diagram illustrating a detection process of a range of a main object according to the third embodiment. FIG. 6 is a diagram illustrating compression of three-dimensional data in the depth direction according to the third embodiment. 7A and 7B are diagrams illustrating a size correction process in a depth direction and a vertical plane direction according to a third embodiment. FIG. 6 is a diagram illustrating a method of detecting a main object range according to the third embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. An example in which the present invention is applied to a digital camera having a configuration capable of measuring a three-dimensional shape of a subject (also referred to as 3D scanning) will be described below as an example of an image processing apparatus. However, the image processing device according to the present invention is not limited to the configuration capable of measuring the three-dimensional shape of the subject, and can be applied to any electronic device capable of acquiring the measured three-dimensional shape data. These electronic devices may include, for example, a mobile phone, a game machine, a tablet terminal, a personal computer, a clock-type or eyeglass-type information terminal, and the like.

(Structure of digital camera 200)
FIG. 2 is a block diagram showing a functional configuration example of a digital camera 200 as an example of the image processing apparatus of this embodiment. Note that one or more of the functional blocks illustrated in FIG. 2 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. May be. Also, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the main subject of operation, the same hardware can be mainly implemented.

The image pickup unit 201 includes a plurality of image pickup optical systems and image pickup elements, and outputs parallax images obtained by the respective image pickup optical systems and image pickup elements. The image pickup device provided inside the image pickup unit 201 receives the subject optical image formed by one image pickup optical system by a plurality of photoelectric conversion elements forming a plurality of two-dimensionally arranged pixels, and receives each pixel. Photoelectrically converted to and output an analog signal. The image pickup device may be an image pickup device such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor outputs an analog signal for each pixel according to the timing signal supplied by the timing generator. The analog signal of each image output from each image sensor is A/D converted and output as digital image data having parallax (parallax image data). The control unit 206 acquires parallax information using the parallax image data output from the imaging unit 201, and based on the parallax information, three-dimensional shape data of an object as a subject (hereinafter, also simply referred to as three-dimensional data). To calculate. It should be noted that one image sensor may be used to configure, for example, each pixel of all effective pixels by a plurality of independent photodiodes to receive the light fluxes of a plurality of different pupil regions of the imaging optical system. Image data corresponding to light fluxes in different pupil regions are image data having parallax with each other, and like the parallax image data, data in the depth direction can be acquired from a pair of image signals corresponding to different pupil regions. .. Further, with this configuration, parallax image data can be obtained using one imaging optical system.

The screen display unit 202 displays the image output from the image capturing unit 201 during the acquisition of the three-dimensional data, and confirms the information (already scanned information) regarding the already acquired shape data, the captured image, and the like.

The screen operation unit 203 is arranged, for example, on the display surface of the screen display unit 202, and when detecting, for example, a touch operation by the user, notifies the control unit 206 of the detection of the operation instruction. The control unit 206 controls the screen display unit 202 in order to change the display such as the menu display in response to the notification from the screen operation unit 203.

The object detection unit 204 performs an object detection process described below on the image data obtained by the image pickup unit 201. The object detection processing performs calculation processing for detecting a person or an object inside the image and specifying their position, size, etc., and outputs the calculation result to the control unit 206.

The button operation unit 205 detects an operation instruction from the user with respect to the operation button or the like, and notifies the control unit 206 of the detection of the operation instruction or the detection content thereof. The control unit 206 controls each unit of the digital camera 200 according to the notification from the button operation unit 205.

The control unit 206 is, for example, a CPU or MPU, and controls the entire digital camera 200 by expanding a program stored in the ROM 207 into a work area of the RAM 208 and executing the program. The control unit 206 calculates three-dimensional data based on the parallax information obtained from the parallax image data, and performs conversion processing of three-dimensional data, which will be described later, on the calculated three-dimensional data.

The ROM 207 is a non-volatile storage device including a semiconductor memory, a magnetic disk, and the like, and stores a program executed by the control unit 206 and setting values set in the digital camera 200.

The RAM 208 is a volatile storage device that stores temporary data that may be erased after the digital camera 200 is powered off. The RAM 208 is responsible for storing temporary data being processed.

The memory slot 209 is an interface for reading/writing recorded data from/to the memory card 210. The memory card 210 is a recording medium composed of magnetism or a semiconductor, and captured images, three-dimensional data, and setting values for the digital camera 200 are recorded via the memory slot 209.

The acceleration sensor 211 and the gyro sensor 212 detect movement and rotation of the digital camera 200 itself. The control unit 206 provides information on the detected movement amount and rotation amount and the object shape information in one direction that can be calculated from the output of the imaging unit 201 (that is, three-dimensional information obtained from the parallax image data) from many viewpoints. Three-dimensional object data can be generated by combining the acquired information. In the present embodiment, parallax image data is used as a means for acquiring information in the depth direction (Z-axis direction), but the present invention is not limited to this, and information in the depth direction such as a TOF sensor using infrared rays or the like is used. The method of acquiring may be another method.

The functional blocks of the imaging unit 201 to the gyro sensor 212 are connected to each other by a bus and can send and receive data.

(Necessity of conversion processing of three-dimensional data according to this embodiment)
With reference to FIG. 1, an example of a problem in the case where three-dimensional data obtained by performing a 3D scan on a space formed by a subject is output as a modeled object will be described. FIG. 1A shows the positional relationship between the digital camera 200 and the physical objects 101 to 104 which are subjects. The objects 101 to 104 exist at respective positions in the depth direction (Z-axis direction) from the digital camera 200, and each has a depth (thickness) peculiar to the object. The object 101 and the object 102 are located at predetermined distances from the Z axis in the plus X direction and the minus X direction, respectively, and are located at equal distances from the digital camera 200 in the Z axis direction. Objects 103 and 104 are located farther (plus Z-axis direction) than the objects 101 and 102 with a predetermined distance. It is assumed that the digital camera 200 performs 3D scanning on the three-dimensional data of the objects 101 to 104. Here, as in the present embodiment, two-dimensional data in the X-axis direction and the Y-axis direction is acquired by the image signal acquired by the image sensor having the photoelectric conversion elements arranged in two dimensions, and further depth information is acquired. The three-dimensional data generated by the configuration will be described. For example, when an image is captured by the image sensor of the image capturing unit 201, the size of the subject in the captured image is smaller than the actual size according to the distance from the digital camera 200. Therefore, even if two-dimensional data (image data) obtained from the image pickup unit 201 is simply added to the Z-axis direction data of the depth information to generate three-dimensional data, the relationship of the size of each subject in the real space is reproduced. It's not always done. In order to model three-dimensional data in the real space, it is necessary to change the size of each subject in the XY directions based on the depth information. However, in the present embodiment, it is assumed that a user who has data that can be viewed as two-dimensional image data outputs a modeled object in order to make the image three-dimensional. Therefore, the three-dimensional data provided to the 3D output device does not necessarily have to be the three-dimensional data that reproduces the relationship of the size of each subject, but rather the relationship of the size of the object (subject) in the XY axis directions. In some cases, the former three-dimensional data that is equal to the image data may be preferable. Therefore, in this embodiment, both of the XY direction data can be considered as the three-dimensional data describing the space including the object.

Regarding the three-dimensional data obtained by the above-described 3D scan, the information in the depth direction will be further considered. The depth information of the obtained three-dimensional data is scaled (normalized) to a size that can be output by the 3D output device, for example, as shown in FIG. The objects 109 to 112 in the three-dimensional data correspond to the objects 101 to 104, respectively. Each object illustrated in FIG. 1B is uniformly reduced in the Z-axis direction according to the range of the 3D output device, and is reduced while maintaining the positional relationship in the depth direction between the objects 101 to 104.

However, when outputting the photographed three-dimensional data as a memorial object or the like, for example, as shown in FIG. 1C, the scale in the depth direction is appropriately reduced in accordance with the features of the object, or in FIG. As described above, there is a need for outputting as a relief-shaped 3D output object. Further, in the output of the three-dimensional shape data of the type shown in FIG. 1B, when the space between the object 111 and the object 112 is wasted, or when the space existing between the object 111 and the object data such as the object 111 exists. In some cases, the important subject is not emphasized and does not look impressive, and the configuration may not meet the user's intention. Therefore, it is not always necessary to accurately reproduce the positional relationship and depth of each object, and it is desirable to perform conversion processing of three-dimensional data that changes the depth between objects as necessary.

Therefore, in this embodiment, for example, the scale of data at a position in a predetermined direction such as the z-axis direction is changed, and the scale (compression ratio) can be changed according to the position (range) in the z-axis direction. That is, in the present embodiment, the data of the position in the predetermined direction of the three-dimensional data is converted according to the position where the object exists. For example, as shown in FIGS. 1C and 1D, the scale of the depth portion between the object 111 and the object 112 is made larger than the scale of the range of the objects 111 and 112. That is, in this embodiment, for example, the scale of the depth direction is changed according to the position in the depth direction by the conversion process of three-dimensional data described below. The scale reduction in the z-axis direction may be performed independently of the scale reduction in the x-axis direction or the y-axis direction orthogonal to the z-axis direction, or may be associated with some scale.

(A series of operations related to conversion processing of three-dimensional data)
Next, with reference to FIG. 3, a series of operations relating to conversion processing of three-dimensional data by the digital camera 200 will be described. This process is started, for example, when a user's shooting instruction to the button operation unit 205 of the digital camera 200 is detected. It should be noted that each step related to this processing is executed in response to an instruction from the control unit 206, and is realized by the control unit 206 expanding a program stored in the ROM 207 into a work area of the RAM 208 and executing the program.

In step S<b>301, the control unit 206 performs a 3D scan that acquires the parallax image data output from the imaging unit 201, the output of the acceleration sensor 211, and the output of the gyro sensor 212, and a shaping process of three-dimensional data based on each acquired data. To do. In the present embodiment, a case will be described as an example where the objects 101 to 104, which are the subjects illustrated in FIG. 1A, are targets of 3D scanning. In this embodiment, an image at a certain position and direction (for example, the digital camera 200 in FIG. 1A) is used as a reference, and scanning is performed from multiple directions, that is, imaging is performed multiple times from multiple directions. At the time of shooting, changes in the shooting position and orientation of the digital camera 200 are acquired in association with each shooting, and the three-dimensional shape of the subject is obtained from the obtained data. A known method can be used as a method of obtaining the three-dimensional shape of an object, which is a subject, based on a plurality of pieces of parallax image data and variations in the shooting position and orientation. For example, by using the information of the acceleration sensor 211 and the gyro sensor 212, it is possible to grasp the positional relationship between a plurality of positions at which the image is taken. Therefore, all the point cloud data as 3D data are located in one three-dimensional space. Can be dropped into. By this combining operation, 3D point cloud data of the surface shape including the front and back of the subject can be acquired. After scanning the three-dimensional shape of the object shown in FIG. 1A, the control unit 206 performs a three-dimensional data shaping process, that is, a process of shaping into a general three-dimensional data such as an STL file format.

The STL file format uses a method of expressing a three-dimensional shape as a set of triangular surfaces, and it is necessary to convert the three-dimensional point cloud data into triangular set data. Although several methods are generally known for this conversion, the three-dimensional Delaunay division method, which is a widely known algorithm, is used in this embodiment. An algorithm of this Delaunay division method is stored in advance in the ROM 207, and the point cloud data of the object obtained by the above-mentioned procedure is converted into a triangular plane set, which is converted into a triangle in a three-dimensional space. It is converted to the STL file format which is the aggregate data of.

In the present embodiment, it is assumed that the molding process in this step changes the scale to a size that can be performed by the 3D output device and generates three-dimensional data in which the x, y, and z axes are reduced to the same scale. explain. When the data shaping process is completed, the control unit 206 stores the shaped data in the memory card 210 via the memory slot 209. In the present embodiment, a 3D scan using a plurality of imaging optical systems and their parallax image data will be described. However, as long as three-dimensional data can be obtained, a mode using laser irradiation, a mode using an image plane phase difference sensor, etc. The form of is also applicable. When the image plane phase difference detection mechanism is mounted on the image pickup unit 201, it is possible to acquire distance data in pixel units. Therefore, the image data obtained by performing the imaging process from a specific position includes color data for each pixel in the image, and in addition, for each pixel, the distance from the digital camera 200 to the surface portion of the subject corresponding to each pixel. Information can be obtained. Point cloud data in which a distance in the depth direction is arranged for each pixel is raw 3D data that can be acquired.

In S302, the control unit 206 determines a coordinate axis to define the depth direction. In this embodiment, for example, the z direction is defined as the depth direction. The y direction represents a vertical direction with respect to the ground, and is defined by identifying the direction of gravitational acceleration by the acceleration sensor 211 during 3D scanning. Since the digital camera itself faces various directions during the 3D scan, it is not determined which is the z-axis direction immediately after the scan. Therefore, the control unit 206 displays the outline of the three-dimensional data of the scan result on the screen display unit 202, and determines the z-axis direction which is the depth direction by the user via the user interface of the screen operation unit 203. When the z-axis direction is determined, the y-axis direction is determined according to the definition of the y-axis, and further the x-axis direction is determined. The control unit 206 converts the three-dimensional data generated in S301 into the three-dimensional data of the newly determined coordinate system and stores it in the memory card 210. In the following description, it is assumed that the z axis shown in FIG. 1A is selected.

Further, among the two-dimensional captured images stored during the 3D scan, an image corresponding to a predetermined cross section of the three-dimensional data is determined. That is, one image that is captured in the determined z-axis direction and in which the above-described objects 101 to 104 are within the angle of view is determined and stored in the memory card 210 as additional information of three-dimensional data. ..

In the present embodiment, since it is premised that scanning (that is, photographing) from multiple directions, the user sets the z-axis direction. However, in the case of a scanning method such as only scanning from one direction and not obtaining back surface information, the scanning direction may be set as the z axis. In this case, the user does not need to specify the z axis.

In S303, the control unit 206 determines a main object to be noticed among the objects which are the subjects. FIG. 4A shows an example of the two-dimensional image determined in S302 and the detection result of the object in the image. The object detection unit 204 detects an object in the image in the two-dimensional image according to an instruction from the control unit 206, and then the control unit 206 displays the detection result of the object detection unit 204 on the screen display unit 202. Since a known object detection technique can be used for detecting an object in a two-dimensional image, detailed description of the object detection technique will be omitted. In the display example in FIG. 4A, object detection frames 405 to 408 for clearly indicating that each of the four objects in the detected image, that is, the objects 401 to 404 are detected are displayed. There is. The user specifies the main object by selecting one of the object detection frames 405 to 408 by an instruction operation on the screen operation unit 203. When a predetermined object is specified by a user operation, the control unit 206 determines the specified object as a main object. Regarding the method of determining the main object, the control unit 206 may automatically determine the main object in addition to the manual determination method by the user described above. In this case, it may be determined based on information such as the position of the object within the angle of view and the characteristics of the object such as whether the object is a person.

In S304, the control unit 206 detects the thickness of the determined main object in the depth direction. The position and orientation of the image shown in FIG. 4A can be obtained from the positional relationship recorded during the 3D scan, and is the position of the digital camera 200 and the orientation of the optical axis shown in FIG. 4B. I shall. The control unit 206 can specify what relationship the object detection frame 407 in FIG. 4A has in the acquired three-dimensional data. More specifically, FIG. 4B shows the position of the object detection frame 407 in the three-dimensional data. The control unit 206 scans the object detection frame 407 in the z-axis direction, and detects three-dimensional data of the object region included in the rectangular parallelepiped 410 formed by the scanning.

More specifically, in the case of the STL file format or the like, the three-dimensional data is represented as a set of three-dimensional triangular information of a minimum unit called a polygon. The control unit 206 first detects all polygons included in the above-mentioned rectangular parallelepiped. Further, in consideration of the connection state between the detected polygons, whether the polygon group constitutes a closed curved surface in the above-mentioned rectangular parallelepiped 410, and the mapping of the range constituted by the closed curved surface onto the xy plane is performed. It is determined whether the object detection frame 407 occupies a certain amount or more. When both of the determinations are satisfied, the control unit 206 determines the three-dimensional data included in the rectangular parallelepiped 410 as the object area of the main object. On the other hand, when the above conditions are satisfied and three or more objects formed of polygonal closed curved surfaces are detected, three-dimensional data relating to one object is determined by adding a condition such as “in front of”. To do. In the case of the present embodiment, the polygon information forming the object 103 can be extracted by performing such processing. It is calculated by estimating the thickness of the object 103 by using the difference between the one having the smallest z-axis component and the one having the largest z-axis component among the detected polygon information constituting the object 103. The calculated thickness becomes the object thickness 409 in FIG.

In S305, the control unit 206 generates a compression profile function H(z) in the z-axis direction based on the position and the thickness of the object 103 in the z-axis direction calculated in S304. Specifically, the compression profile function H(z) is a function of z indicating at what position on the z axis the component is compressed and at what scale, and represents a conversion characteristic in the z direction. The compression profile function H(z) is generated using the position and thickness of the object 111 in the z-axis direction calculated in S304. With reference to FIG. 5, the compression in the z-axis direction including the generation processing of the compression profile function H(z) will be described. In the following description, the z direction will be described for simplicity, but the x and y directions will be converted at a compression rate of 1, for example.

FIG. 5A schematically shows the positional relationship in the three-dimensional data as viewed from the direction perpendicular to the z axis (the positive side of the x axis). Further, FIG. 5B shows the characteristics of the compression profile function H(z), the horizontal axis shows the depth (z), and the vertical axis shows the compression rate. For example, when the compression rate is 1, the z-axis direction component is maintained as it is. Further, for example, when the compression rate is 2, the distance in the z-axis direction is further compressed to 1/2. As for the object 111, since the compression ratio of the portion corresponding to the position where the object 111 exists and the thickness range is 1, the position and the thickness do not change. On the other hand, for the positions other than the object 111, the compression rate linearly increases with the distance from the position. Therefore, the positional relationship between the object 111 and the objects 109 and 112 in the z-axis direction is compressed, and as shown in FIG. 5C, between the objects 501 and 502 and between the objects 502 and 503. Is shortened like the distance. Further, since the thickness of each of the objects 109 and 112 is also compressed, each is compressed to the thickness of each of the objects 501 and 503 shown in FIG. 5C. In this way, the control unit 206 keeps the compression rate low for the main object based on the position and thickness of the main object in the z-axis direction, and linearly increases the compression rate according to the distance from the main object (that is, monotonically increases). , The compression profile function is generated.

In S306, the control unit 206 generates a conversion function G(z) for actually converting the z-axis data of the three-dimensional data from the compression profile function. Expressed by a continuous system formula, the conversion function G(z) can be generated from the compression profile function H(z) by the formula represented by Formula 1. The mathematical expression 1 is a mathematical expression represented by the mathematical expression 2 in a discrete system mathematical expression suitable for CPU calculation and the like.

The compression profile function H(z) in the z-axis direction, the conversion function G(z), and z'after conversion will be described with reference to FIG. For simplification of description, the horizontal axis z is assumed to be normalized to 0-100. The position of z=0 corresponds to the forefront position of the data range in the z-axis direction, and the position of z=100 corresponds to the rearmost position of the data range in the z-axis direction. FIG. 5D shows the compression profile function H(z), and the compression rate other than the range (z=30 to 40) of the main object can be set arbitrarily. The compression rate outside the range of the main object may be calculated from the relationship with the position and the thickness of the main object, for example, with the compression rates at the frontmost position and the rearmost position in the z-axis direction as the predetermined compression rates. Good. The conversion function G(z) shown in FIG. 5(e) is a function calculated by applying Equations 1 and 2 to the compression profile function H(z) shown in FIG. 5(d). When the coordinate before compression conversion is z and the coordinate after conversion corresponding to z is z′, the relationship between z and z′ is expressed by Equation 3. FIG. 5(f) is a graph showing the relationship between z and z'.

In step S307, the control unit 206 performs a process of converting the z-axis direction data of the three-dimensional data obtained in step S302 using the z-axis conversion function G(z) determined in step S306. FIG. 6 shows a syntax of an STL (STereoLithography) file format which is an example of a file format of three-dimensional data. As shown in FIG. 6, the three-dimensional data is represented in a format in which polygon information, which is triangular information in a three-dimensional space, is listed. More specifically, the "facet normal" line to the "endfacet" line form concrete data of the triangle data, and the concrete data is described in the "facet normal" line and the "vertex" line. The "facet normal" row represents the normal vector of the triangle, the three "vertex" rows represent the coordinates of each vertex of the triangle data, and each vertex has a numerical value of x, y, and z components. Therefore, the control unit 206 converts the position in the z-axis direction by applying the conversion function G(z) to the information on each vertex of the triangle.

FIG. 7 shows how the polygon triangle information is converted in the z-axis direction using the z-axis conversion function G(z). The polygon information indicated by S in FIG. 7A is composed of three vertices of A (x1, y1, z1), B (x2, y2, z2) and C (x3, y3, z3). The control unit 206 does not change the x component and the y component with respect to all the polygon information included in the three-dimensional data, calculates z′ shown in Expression 3 only for the z component, and changes z by z′. Create the data that you created. FIG. 7B shows polygon information in which S shown in FIG. 7A is S′ by compressing only the z component by Expression 3. The respective vertex information of A to C constituting S is converted into A'(x1, y1, z1'), B'(x2, y2, z2'), C'(x3, y3, z3'), respectively. There is.

When the conversion processing for all the polygon information is completed, the control unit 206 stores the converted polygon data listed in the STL format in the memory card 210 via the memory slot 209. When the control unit 206 stores the converted three-dimensional data, the control unit 206 ends the series of operations related to the conversion process of the three-dimensional data.

In this embodiment, the three-dimensional data obtained by the 3D scan in S301 is reduced in advance in the same scale in the x, y, and z directions, and then the conversion process in the z-axis direction is performed. did. However, in S301, the scale obtained based on the compression profile function, which is different from the x-axis direction and the y-axis direction, may be applied to the z-axis direction without performing the reduction by the same scale in advance. Good. In this case, if the compression profile function H(z) is a function representing the scale of the real object, the processing of this embodiment can be realized.

Further, in the present embodiment, in the compression profile function H(z), the compression ratio is linearly changed in the region other than the main object to convert the depth direction of the three-dimensional data. However, the characteristics of the compression profile function H(z) can be various characteristics according to the user's intention or the characteristics of the object. For example, modified examples of H(z) are shown in FIGS. 8A is the same as FIG. 5A described above, but FIG. 5B determines how much the distance in the z-axis direction of the three-dimensional data shown in FIG. 8A is compressed. The characteristic of the compression profile function H(z) is shown. In H(z) shown in FIG. 8B, the compression ratio is increased in a stepwise manner from the end position on the z axis of the main object. By doing so, as shown in FIG. 8C, while maintaining the positional relationship between the converted object 901 and the object 902 (main object), the object 903 on the inner side of the main object can be The depth can be reduced significantly. Moreover, another modification is shown in FIGS. 8(d) is the same as FIG. 8(a), but FIG. 8(e) shows the start of the main object on the z-axis in the compression profile function H(z) shown in FIG. 5(b). Before and after the position and the end position, the compression ratio is raised and lowered in a stepwise fashion. In this way, the positional relationship and the depth of the object 1002 (main object) before and after the object 1002 (main object) are reduced at the same magnification while maintaining the thickness of the converted main object. In this case, the main object can be emphasized while matching the distance between the objects 1001 and 1002 and the distance between the objects 1002 and 1003 with the positional relationship of the real objects.

As described above, in the present embodiment, the scale for the data at the position in the predetermined direction such as the z-axis direction is changed, and the scale (compression rate) can be changed according to the position (range) in the z-axis direction. .. That is, the compression profile function H(z) is determined in the depth direction, and the data of the position in the depth direction of the three-dimensional data is converted according to the position where the object exists. By doing so, it becomes possible to transform the three-dimensional data for outputting the modeled object at a scale or a double scale suitable for the modeled object.

Also, when changing the scale according to the position in the depth direction, depending on the type of object, that is, while reducing the compression rate in the depth direction with respect to the area of the main object, increase the compression rate in areas other than the area of the main object, The depth of the 3D data is modified. That is, the overall compression rate in the depth direction and the compression rate in the depth direction of a specific subject are set. By doing so, the main object can be emphasized or given a presence as compared with other objects as shown in FIG. 5C, and a modeled object suitable as a memorial object or an interior object can be obtained. It will be possible to generate.

(Embodiment 2)
Next, a second embodiment will be described. In the first embodiment, the three-dimensional data conversion process is performed using the compression profile function that lowers the compression rate of the selected main object and increases the compression rate of the other area. In the present embodiment, conversion processing of three-dimensional data is performed using a compression profile function with a reduced compression rate for a plurality of main objects, that is, sub-main objects other than the main object. For this reason, the processing for determining the sub main object and the determined compression profile function are different from those of the first embodiment, and the other configurations are the same as those of the first embodiment. For this reason, the same configurations and steps are denoted by the same reference numerals, duplicate description is omitted, and different points will be mainly described.

The flow of processing in this embodiment will be described with reference to the flowchart shown in FIG.

First, the control unit 206 performs the processing of S301 to S303 shown in FIG.

Next, in step S901, the control unit 206 determines a sub-main object whose priority is lower than that of the main object selected in step S303, but which has high priority. As shown in FIG. 4A, with respect to the four objects in the image detected by the process in S303, that is, the objects 401 to 404, the object detection frames 405 to clearly indicate that they have been detected. 408 is displayed. The user operates the screen operation unit 203 to select one of the object detection frames other than the main object selected in S303. Upon receiving the user operation notification from the screen operation unit 203, the control unit 206 determines the corresponding object as the sub-main object. Here, it is assumed that the control unit 206 determines the object 112 as the sub-main object in response to the user selecting the object detection frame 408. Note that the sub-main object may be automatically determined by the control unit 206 instead of being manually selected by the user. In this case, the sub-main object may be determined by determining where the object is within the angle of view and whether the object is a person. If necessary, a plurality of sub-main objects may be designated.

In S902, the control unit 206 calculates the thickness in the z-axis direction for each of the main object and the sub-main object. The control unit 206 calculates the thickness of each of the main object and the sub-main object in the same manner as the method described in S305. For example, as shown in FIG. 10, the control unit 206 scans the object detection frame 408 of the sub main object in the z-axis direction, and detects the three-dimensional data included in the rectangular parallelepiped body 1010 formed by the scanning. At this time, when another rectangular parallelepiped such as the rectangular parallelepiped 410 is detected, the rectangular parallelepiped having the highest ratio of the area of the xy plane of the rectangular parallelepiped to the object detection frame 408 may be selected.

In S903, the control unit 206 generates the compression profile function H(z) based on the positions and thicknesses of the main object and the sub-main object in the z-axis direction calculated in S902. The compression profile function H(z) according to this embodiment will be described with reference to FIG.

FIG. 11A schematically shows the positional relationship of the objects viewed from the direction perpendicular to the z axis (plus x axis direction). FIG. 11B is a compression profile function that determines how much the z-axis direction information of the object shape data similar to that in FIG. 5B is compressed. In FIG. 11B, the compression ratio is set to 1 for the position where the main object exists and the thickness range, as in FIG. 5B. On the other hand, with respect to the distance corresponding to the position where the object 112, which is the sub main object, exists and the thickness range, the compression rate is uniformly set to 2. Except for the positions of the objects 111 and 112 and the thickness region, the compression rate is set to increase linearly with the distance from the position of the object 111.

As described above, in the present embodiment, the control unit 206 sets such that the compression rate for the main object is low and the compression rate increases linearly according to the distance from the main object, but exceptionally for the sub-main object. The compression profile function H(z) is generated so that the compression rate becomes low. By doing so, as shown in FIG. 11C, in the reduced z′, in addition to maintaining the thickness of the object 1102 (main object), the thickness of the object 1103 (sub-main object) is also reduced. It has been maintained and reduced. That is, the positional relationship between the objects 1101 to 1103 is reduced, but the main object and the sub-main object are relatively large, and the existence of these objects is emphasized. Since the compression rate for the sub-main object is set to a compression rate different from that of the main object, the sub-main object is reduced at a slightly higher scale than the main object, and the main object is emphasized more than the sub-main object. Has been adjusted.

Next, the control unit 206 performs the processes of S306 and S307 shown in FIG. 3 and ends the series of processes related to this process. It should be noted that FIG. 11E shows a conversion function G(z) generated by Equation 1 and Equation 2 from the compression profile function H(z) shown in FIG. 11D generated by the above method. .. In addition, FIG. 11F shows the relationship between z converted by the conversion function G(z) and z′ after conversion.

In the present embodiment, the compression profile function H(z) is set to linearly increase the compression rate in the range other than the main object. However, it is stepwise from the end position on the z axis of the main object and the sub-main object. The compression rate may be increased. Alternatively, the compression rate may be automatically generated in a stepwise manner before and after the start position and the end position on the z axis of the main object and the sub main object.

Further, in the compression profile function H(z), the compression ratio in the range of the main object or the like has been described as a predetermined value. However, the scale (or compression rate) in the depth direction may be set or changed according to the position by inputting to the user interface of the screen operation unit 203 or the like. By doing so, it is possible to generate a modeled object in which the main object is emphasized at a scale desired by the user.

As described above, in the present embodiment, the conversion processing of three-dimensional data is performed using the compression profile function in which the compression rates of the main object and the sub-main object are reduced as shown in FIG. .. By doing so, not only the main object but also the sub-main object can be emphasized or given a presence, and it becomes possible to generate a modeled object suitable as a memorial object or an interior object.
(Embodiment 3)
Next, a third embodiment will be described. In the above-described embodiment, the three-dimensional data conversion process is performed using the compression profile function that lowers the compression rate of the selected main objects (main object and sub-main object) and increases the compression rate of other areas. It was In the present embodiment, in the conversion process of three-dimensional data, a process of correcting the size in the xy plane direction of a part of a plurality of main objects (for example, sub main objects) is further performed. Therefore, the processes and configurations other than the correction process of the size are the same as those in the second embodiment. For this reason, the same configurations and steps are denoted by the same reference numerals, duplicate description is omitted, and different points will be mainly described.

First, the relationship between the size and the position of the main object and the sub-main object assumed in this embodiment will be described with reference to part of FIG. 13A (details of FIG. 13A will be described later). In the present embodiment, it is assumed that the main object 1301 and the sub-main object 1302 have an actual size of the main object 1301 that is, for example, half the actual size of the sub-main object 1302. For example, the main object 1301 is a child and the sub-main object 1302 is an adult. It is also assumed that the distance (L1) from the digital camera 200 to the main object 1301 is half the distance (L2) from the digital camera 200 to the sub-main object 1302. In this case, the two-dimensional image in the xy directions taken from the position of the digital camera 200 is as shown in FIG. That is, although the actual size of the main object 1301 is half that of the sub-main object 1302, it becomes the same size on the two-dimensional image. If the final three-dimensional shaped object is output at this ratio, this object may not be viewed from the direction and position shown in FIG. 13A, and the size relationship may be unnatural. There is a nature. Therefore, in the present embodiment, the size of the sub-main object in the xy plane direction is normalized (that is, corrected) with reference to the size of the main object in the xy plane direction. Hereinafter, in the present embodiment, a correction process for generating a modeled object in which the sizes of an adult and a child are more appropriately reflected will be described as an example.

(A series of operations related to conversion processing of three-dimensional data)
Next, a series of operations of the conversion processing of three-dimensional data according to the present embodiment will be described with reference to the flowchart shown in FIG.

First, the control unit 206 performs the processing of S301 to S303 shown in FIG.

In step S1201, the control unit 206 identifies a sub-main object having a higher priority than the main subject selected in step S303 but having a lower priority. The process of this step is the same as the process of S901 described above, but it will be described with reference to FIG. In the two-dimensional image selected in S302, the two objects (1301 and 1302) detected in S303 and object detection frames 1303 and 1304 that clearly indicate that each object has been detected are displayed. At this time, when the user operates the screen display unit 202 to select an object detection frame other than the main object selected in S303, the control unit 206 determines the object corresponding to the selected object detection frame as the sub-main object. To do. For example, when the user selects the object detection frame 1304, the sub main object 1302 is determined as the sub main object.

In step S1202, the control unit 206 calculates the thickness in the z-axis direction for each of the main object and the sub-main object. The control unit 206 calculates the thickness in the z-axis direction in the same manner as the method shown in S305. For example, as shown in FIG. 13B, the control unit 206 scans the object detection frame 1303 for the main object 1301 (object 1307) and the object detection frame 1304 for the sub main object 1302 (object 1308) in the z-axis direction. To do. Then, the three-dimensional data included in the rectangular parallelepiped 1309 and 1310 formed by scanning is detected to detect the thicknesses 1305 and 1306 of the object.

In step S1203, the control unit 206 generates a compression profile function H(z) based on the position and the thickness in the z-axis direction with respect to the object 1307 and the object 1308 calculated in step S1202. The compression profile function H(z) according to this embodiment will be described with reference to FIGS. 14A, 14B, and 14D to 14F as appropriate.

FIG. 14A schematically shows the positional relationship of the objects viewed from the direction perpendicular to the z axis (plus x axis direction). FIG. 14B shows a compression profile function that determines how much the z-axis direction information of the object shape data is compressed, similar to the compression profile function in the above-described embodiment. In FIG. 14B, the compression ratio is set to 1 for the position and thickness range where the object 1307, which is the main object, exists, as in FIG. 11B. On the other hand, for the distance corresponding to the position where the object 1308 which is the sub-main object exists and the thickness range, the compression rate is set to 2 uniformly. Except for the positions of the object 1307 and the object 1308 and the thickness region, the compression rate is set to increase linearly with the distance from the position of the object 1307.

As described above, in the present embodiment, the control unit 206 sets the compression rate for the main object to be low and the compression rate to increase linearly according to the distance from the main object, while exceptionally setting the sub-main object. The compression profile function H(z) is generated so that the compression rate becomes low. By doing so, in the reduced z′, in addition to maintaining the thickness of the object 1307 (main object), the thickness of the object 1308 (sub-main object) is also maintained and reduced.

Next, the control unit 206 performs the processes of S306 and S307 shown in FIG. Note that FIG. 14E shows a conversion function G(z) generated by the mathematical formulas 1 and 2 from the compression profile function H(z) shown in FIG. 14D generated by the above method. .. In addition, FIG. 11F shows the relationship between z converted by the conversion function G(z) and z′ after conversion.

Further, in S1204 and S1205, the control unit 206 corrects the size of the sub main object in the xy plane direction with respect to the main object.

First, in step S1204, the control unit 206 acquires the setting by the user who has selected whether or not to correct the size of the object 1308 (sub-main object) in the xy plane direction. This setting is for selecting whether or not to correct the size of the object 1308 (sub-main object) in the xy plane direction according to the size of the object 1307 (main object). The control unit 206 advances the processing to S1205 when the acquired setting is the setting for performing the correction, and ends the series of operations related to the present processing without performing the correction when it is the setting for not performing the correction.

In step S1205, the control unit 206 corrects the size of the object 1308 (sub main object). More specifically, the control unit 206 calculates a parameter for correcting the size of the object using the ratio of the distance to each object. For example, FIG. 15 shows a schematic view of the yz plane when FIG. 13B is viewed from the x-axis direction. For example, it is assumed that the distance from the digital camera 200 to the main object 1301 is L1, the distance from the digital camera 200 to the sub-main object 1302 is L2, and there is a relationship of L2=L1×2. Further, it is assumed that the length of the main object 1301 in the y direction is H1, the length of the sub main object 1302 in the y direction is H2, and there is a relationship of H2=H1×2. Further, the angle of view of the digital camera 200 is θ. At this time, if the vertical width corresponding to the range of the angle of view at the distance of L1 is the vertical angle of view 1 and the vertical width corresponding to the range of the angle of view at the distance of L2 is the vertical angle of view 2, the similar relationship is obtained. Therefore, the view angle vertical width 2 is twice the view angle vertical width 1. Therefore, the size of each object in the two-dimensional image when taken from a certain imaging position is inversely proportional to the distance to the object. That is, the control unit 206 corrects (that is, scales) the size in the x direction and the y direction of the 3D data information of the object 1308 (sub-major object) using the ratio of L2/L1, and the sub-major of the object 1402. Convert to object data. For example, FIG. 14C shows an object 1401 and an object 1402 which are the result of the size correction performed on the object 1308.

By doing so, it is possible to make the size of the converted object 1401 and the converted object 1402 in the xy plane directions more realistic. Further, as in the second embodiment, the sub-main object data of the object 1402 also becomes a model with a thickness emphasized as compared with the object (for example, 503) after the conversion by the method of the first embodiment.

Then, when the control unit 206 generates the sub main object data of the object 1402, the control unit 206 ends the series of operations related to this processing.

Note that in S1204 described above, an example has been described in which whether or not to correct the size of the sub main object is selected and set. As described above, the reason why correction is to be performed is selectable because it may be more appropriate to turn correction off. For example, in the example shown in FIGS. 16A and 16B, the main object 2001 and the sub-main object 2002 are photographed as subjects in the two-dimensional image, and the corresponding three-dimensional object 2003 (main object) and An object 2004 (sub-main object) is shown.

In this example, the main object 2001 is a person, and the sub-main object 2002 is a huge landmark such as a tower. At this time, it is assumed that the distance L4 from the digital camera 200 to the object 2003 is much longer than the distance L4 from the object 2004. Further, the image shown in FIG. 16A has a composition in which the entire main subobject main body 2002, which is a tower, is projected together with the main main object 2001, which is a person, such as a commemorative photo. When the correction process described in S1205 is performed in such a case, the 3D data of the sub main object 2002 is enlarged by the actual ratio and becomes excessively large, and the relationship between the subjects intended when the image of FIG. May collapse. Therefore, it is desirable to be able to perform control so that the size of the sub-main object is not corrected (that is, turned off) when the image capturing shown in FIG. 16 is performed.

That is, according to the present embodiment, it is possible to control the presence/absence of the correction processing of the sizes of the main object 1301 and the sub-main object 1302 in the xy plane direction according to the shooting content. Even if this correction process is not performed, as shown in FIG. 11(c), the main object and the sub-main object are emphasized, and the parts other than the object are compressed, so that a souvenir, an interior object, etc. The data can be converted to data suitable for outputting the modeled object.

In the present embodiment, an example in which the sub main object is determined by a user operation in S1201 has been described. However, the control unit 206 may automatically determine one or more sub-main objects based on, for example, the position within the angle of view of the sub-main object or the determination result of whether the object is a person or the like.

Further, in the present embodiment, an example in which the selection of the presence/absence of the correction process is selected by the user operation in S1204 has been described. On the basis of this, the presence/absence of correction processing may be automatically selected. For example, the control unit 206 may turn off the correction process when the difference in size between the main object and the sub-main object is larger than a predetermined threshold. Further, for example, the control unit 206 may turn on the correction process when it is determined that both the main object and the sub-main object are persons.

As described above, in the present embodiment, the size of the sub main object in the xy plane direction is corrected based on the ratio of the distance between the main object and the sub main object in the depth direction. By doing so, with respect to the three-dimensional shape data for outputting the modeled object, the size in the direction perpendicular to the depth direction (xy plane direction) is converted to a more realistic size relationship and output. be able to. Further, whether or not to perform this correction can be set, and in particular, when the ratio of the distance in the depth direction between the main object and the sub-main object does not have an appropriate relationship, the correction is not performed. By doing so, it becomes possible to transform the three-dimensional shape data for outputting the modeled object at a scale or a double scale suitable for the modeled object.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

201... Imaging unit, 202... Screen display unit, 203... Screen operation unit, 204... Object detection unit, 205... Button operation unit, 206... Control unit

Claims (15)

Acquisition means for acquiring three-dimensional data describing a space including an object,
Conversion means for converting the position data of the three-dimensional data in a predetermined direction according to the position of the object,
Have a, and output means for outputting the three-dimensional data position in the predetermined direction is converted,
When there are a plurality of objects, the conversion means compares the position data in the range corresponding to the space in which the plurality of objects do not exist with the position data in the range corresponding to the space in which the plurality of objects exist. An image processing apparatus, characterized in that conversion is performed at a large compression rate . The image processing apparatus according to claim 1, wherein the conversion unit converts each position in the predetermined direction based on a type of an object existing in the three-dimensional data. With respect to the data of the position in the predetermined direction of the three-dimensional data, the converting means may be more than the data of the position in the range in which the distance from the position corresponding to the space in which the reference object exists is the first distance. 3. The image processing apparatus according to claim 1, wherein data at a position in a second distance that is farther than the first distance is converted with a larger compression rate. And the converting means, according to claim 3, in accordance with the distance from the position corresponding to the existing space of the object to the reference, the compression ratio is characterized by compressing the data of the position to increase monotonously Image processing device. Further comprising first input means for inputting a user operation for the compression ratio of the data at the position in the predetermined direction,
And the converting means, in any one of claims 1 to 4, characterized in that converting the data of position of said predetermined direction of said three-dimensional data based on the input to the first input means operation The image processing device described. The conversion means converts the data of the position in the range corresponding to the space in which the predetermined object exists among the plurality of objects, to the ratio of the distance from the predetermined position to each of the two objects including the predetermined object. The image processing apparatus according to any one of claims 1 to 5, characterized in that the magnification is also changed in a direction perpendicular to the predetermined direction based on the above. Further comprising a determining means for determining whether to change the magnification also in a direction perpendicular to the predetermined direction,
The determining means is a second input means for acquiring a user operation for selecting whether or not to change the magnification also in a direction perpendicular to the predetermined direction, or a vertical direction in the predetermined direction based on the type of the predetermined object. Including a determining means for determining whether to scale in any direction,
When the conversion unit determines to scale in the direction perpendicular to the predetermined direction by the determination unit, the data of the position of the range corresponding to the space in which the predetermined object of the plurality of objects exists. The image processing apparatus according to claim 6 , wherein the image is also magnified in a direction perpendicular to the predetermined direction. The three-dimensional data includes two-dimensional image data obtained from an image pickup device having two-dimensionally arranged photoelectric conversion elements, and data in the depth direction of the subject as position data in the predetermined direction. The image processing apparatus according to any one of claims 1 to 7 , which is characterized. Further comprising a detection means for detecting a specific object from the image data,
The image processing apparatus according to claim 8 , wherein the conversion unit determines an object existing in the three-dimensional data based on a detection result of the detection unit. The converting means determines the range in which the object detected from the image data by the detecting means exists in the predetermined direction of the three-dimensional data based on the thickness information of the object detected by the detecting means which is stored in advance. The image processing apparatus according to claim 9 , wherein the image processing apparatus estimates the image according to the method. The image pickup device receives light fluxes of a plurality of different pupil regions of the image pickup optical system by the plurality of photoelectric conversion elements,
Data of the position in the predetermined direction of the three-dimensional data, according to any one of claims 8 to 10, wherein the those calculated from a pair of image signals corresponding to different pupil areas Image processing device. And the output means, the converted into a predetermined format in order to use the three-dimensional data converted by the conversion means to generate a three-dimensional shaped object, any one of claims 1 to 11, characterized in that output 1 The image processing device according to item. It said output means converts the 3-dimensional data in the STL file format, the image processing apparatus according to any one of claims 1 to 12, characterized in that output. An acquisition step in which the acquisition means acquires three-dimensional data describing a space including an object;
A conversion step in which the conversion means converts the position data of the three-dimensional data in a predetermined direction according to the position of the object;
Output means, have a, and an output step of outputting the three-dimensional data position in the predetermined direction is converted,
In the conversion step, when a plurality of objects are present, the position data in the range corresponding to the space in which the plurality of objects do not exist is compared with the position data in the range corresponding to the space in which the plurality of objects exist. A method for controlling an image processing device, characterized in that conversion is performed at a large compression rate . Program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 13.