JP2012109723A - Image processing device, image processing method, control program for image processing device, and computer readable recording medium with program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize an image processing device that performs processing for reducing (smoothing) sharpness of all or part of an image with a simple configuration.SOLUTION: An image processing device 101 comprises: a distance image data analysis unit 104 that acquires, for each pixel, the pixel value of a pixel constituting distance image data in which a distance to an object captured by a distance image sensor 107 is recorded for each pixel, and compares the acquired pixel value with a predetermined threshold value for each pixel; and an image processing unit 103 that performs a predetermined operation on the pixel value of a pixel constituting captured image data captured by a camera 106 corresponding to a pixel whose distance is determined to exceed the predetermined threshold value as a result of the comparison.

Description

  The present invention relates to an image processing apparatus or the like that performs a process of reducing (blurring) the sharpness of part or all of an image.

  In a general videophone device, an image obtained by photographing a user of the videophone device with a camera is transmitted as it is to the other party's videophone device which is a call destination. However, the user may not want to show the background to the other party when the room is cluttered.

  In such a case, Patent Documents 1 and 2 disclose a technique for performing image processing on an image to be transmitted to the other party's videophone device so that the background of the user is not shown to the other party. Technology.

  Patent Document 1 discloses a technique for extracting a human face, in particular, a facial part from an image to be transmitted, and performing a fade process on the image part other than the extracted part. In the technique disclosed in Patent Document 1, only the face of a person extracted from the image to be transmitted is shown to the other party, and the facial expression of the user is transmitted to the other party. The private image information that appears in the background of the user is not shown to the other party.

  Patent Document 2 discloses a technique for obtaining a distance to an object using images photographed from two cameras and replacing a background at a distance recognized as other than the object to be photographed with an image prepared in advance. It is disclosed.

JP-A-6-70313 (published on March 11, 1994) Japanese Patent Laid-Open No. 6-311405 (released on November 4, 1994)

  In the technique disclosed in Patent Document 1, for example, a person's face is extracted from an image to be transmitted so that only the extracted image portion is shown, and an object other than the face is shown to the other party. Even when it is desired, there arises a problem that the image of the object is erased by the fade processing or the like.

  Here, in the videophone device according to Patent Document 1, in order to solve the above problem, it is conceivable to increase the number of parts to be extracted and include the object in the extraction target. However, in order to increase the number of parts to be extracted, it is necessary to add at least as many image processing circuits and binarization circuits as the number of parts to be extracted. For this reason, in the videophone device according to Patent Document 1, when an object other than the face is shown to the other party, there is a problem that the circuit scale increases.

  In addition, in the technique disclosed in Patent Document 1, when the videophone device is used in a crowd, there is a possibility that other people's faces that pass behind the user may be extracted in addition to the user's face. The other person's face extracted can also be seen from the other party. Therefore, the technique disclosed in Patent Document 1 has a problem that an image that appears in the background of the user and that the user does not want to show to the other party may be seen by the other party.

  In the technique disclosed in Patent Document 2, since the background is replaced with an image prepared in advance, there is a problem in that the situation at the call site may not be communicated to the other party. For example, even if you are calling from home at night, if you use the daytime scenery in the southern country as a background image, you will not be able to tell the other party whether you are actually calling from home.

  Further, the image transmission apparatus according to Patent Document 2 requires a memory for recording a background image, which causes a problem that the circuit scale increases.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus or the like that performs processing to reduce (blur) part or all of an image with a simple configuration. There is to do.

  In order to solve the above-described problem, the image processing apparatus of the present invention performs a process for reducing the sharpness of a part or all of the captured image on the captured image obtained by capturing the captured image. Is a distance image obtained from a distance image sensor, and each of the distance pixels that constitute the distance image is a pixel that constitutes the captured image. Distance image acquisition means that is associated with any one of the imaging pixels and that obtains a distance image in which distance data indicating the distance from the distance image sensor of the imaging target is recorded in association with each distance pixel; The distance data recorded in association with the distance pixel is acquired for each distance pixel from the distance image acquired by the distance image acquisition means, and the acquired distance data is a predetermined threshold value. A determination unit that determines whether or not the distance pixel exceeds the pixel value of the imaging pixel corresponding to the distance pixel that is determined by the determination unit that the distance data exceeds the predetermined threshold among the captured images. And an image generation means for generating the output image by performing a predetermined calculation.

  Further, in order to solve the above-described problem, the image processing method of the present invention performs a process for reducing the sharpness of a part or all of the captured image on the captured image obtained by capturing the captured image, An image processing method for outputting an output image with reduced sharpness, which is a distance image obtained from a distance image sensor, wherein each of the distance pixels that constitute the distance image is a pixel that constitutes the captured image A distance image acquisition step of acquiring a distance image that is associated with any one of the imaging pixels and that indicates distance data indicating a distance from the distance image sensor of the imaging target in association with each distance pixel. And distance data recorded in association with the distance pixel from the distance image acquired in the distance image acquisition step for each distance pixel, and the acquired distance data. A determination step for determining whether or not the distance pixel exceeds a predetermined threshold value, and the imaging corresponding to the distance pixel that is determined in the determination step that the distance data exceeds the predetermined threshold value among the captured images. And an image generation step of generating the output image by performing a predetermined calculation on the pixel value of the pixel.

  According to the above configuration, the image processing apparatus of the present invention compares the distance data recorded in each distance pixel associated with each imaging pixel based on the result of comparison with the predetermined threshold. Whether or not to perform the image processing (predetermined calculation) for reducing the sharpness is controlled with respect to the imaging pixel corresponding to the distance pixel in which data is recorded.

  That is, the image processing apparatus of the present invention performs image processing for reducing (blurring) the sharpness when the distance from the distance image sensor to the imaging target exceeds a predetermined threshold in the pixel unit of the imaging pixel.

  For this reason, for an object (such as a user of an image processing apparatus) that is close to the distance image sensor, if the distance of the object from the distance image sensor does not exceed the predetermined threshold, the object in the captured image is captured. The image processing for reducing the sharpness is not performed on the region.

  On the other hand, when the distance from the distance image sensor of the object at a position far from the distance image sensor (such as the background of the user of the image processing apparatus) exceeds the predetermined threshold, the object is captured in the captured image. Image processing for reducing the sharpness is performed on the region.

  That is, the image processing apparatus of the present invention can generate an image with a reduced sharpness so as to make it difficult to see the background or the like of the user of the image processing apparatus with a simple configuration.

  Further, the image processing for reducing the sharpness can be realized with a simple configuration because a generally known image blurring technique can be applied.

  Further, the image generation means of the image processing apparatus of the present invention performs the predetermined calculation using the pixel value of the imaging pixel to be subjected to the predetermined calculation and the pixel value of the imaging pixel adjacent to the imaging pixel. It is characterized by that.

  Further, the image generation unit of the image processing apparatus of the present invention uses, as the predetermined calculation, a pixel value of an imaging pixel to be subjected to the predetermined calculation, a pixel value of the imaging pixel, and an imaging pixel adjacent to the imaging pixel. The pixel value is changed to a weighted average value.

  Furthermore, the image generation unit of the image processing apparatus of the present invention uses a coefficient corresponding to the distance data recorded in association with the distance pixel corresponding to the imaging pixel to be subjected to the predetermined calculation, as a coefficient of the imaging pixel. The weighted average value is calculated after multiplying the pixel value and the pixel value of the imaging pixel adjacent to the imaging pixel.

  According to the above configuration, the coefficient corresponding to the distance data is multiplied by the pixel value of the imaging pixel to be processed and the pixel value of each imaging pixel in the vicinity thereof to obtain a weighted average value of these pixel values, By changing the pixel value of the imaging pixel to be processed to the weighted average value, image processing for reducing the sharpness is performed on the imaging pixel to be processed.

  According to the above configuration, the image processing for reducing the sharpness can be realized by blurring the captured image using a known blur filter (low-pass filter). An output image in which the state before the blurring process can be determined can be created.

  Further, according to the above configuration, by changing the coefficient according to the distance data, in the captured image, the blurring degree is reduced between an object at a distance close to the distance image sensor and an object at a distance far from the distance image sensor. Can be changed. Thereby, it is possible to perform processing such as blurring an image of an object at a distance far from the distance image sensor more significantly than an image of an object at a distance near the distance image sensor.

  The image processing apparatus of the present invention is further characterized by further comprising image transmission means for transmitting the output image generated by the image generation means to the outside of the image processing apparatus.

  According to the above configuration, since the output image can be transmitted to the outside of the image processing apparatus, the external apparatus can use the output image.

  An example of a device provided with the image processing device of the present invention is a videophone device. The videophone device including the image processing device of the present invention can reduce the possibility that the object is erased from the image by disposing the object to be shown to the other party at a position close to the distance image sensor. In addition, the videophone apparatus including the image processing apparatus of the present invention distinguishes whether or not to perform image processing for reducing the sharpness according to the predetermined threshold without increasing the number of parts (targets) extracted from the image. Therefore, it is not necessary to increase the circuit scale. In addition, since the videophone apparatus provided with the image processing apparatus of the present invention reduces the clearness of the background through image processing, it is not necessary to replace the background, and it is possible to convey the situation of the call site to the other party. It becomes. Furthermore, since the videophone apparatus provided with the image processing apparatus of the present invention does not require a memory for recording a background image, it is possible to suppress an increase in circuit scale.

  The image processing apparatus may be realized by a computer. In this case, an image processing apparatus control program for causing the image processing apparatus to be realized by the computer by causing the computer to operate as the above-described means, and A computer-readable recording medium on which is recorded also falls within the scope of the present invention.

  As described above, the image processing apparatus of the present invention performs processing for reducing the sharpness of part or all of the captured image on the captured image obtained by capturing the imaging target, and outputs the image with the reduced sharpness. A distance image obtained from a distance image sensor, wherein each of the distance pixels that constitute the distance image is one of the imaging pixels that constitute the captured image. Distance image acquisition means for acquiring a distance image in which distance data indicating the distance from the distance image sensor to be imaged is associated and recorded for each distance pixel, and the distance image acquisition means The distance data recorded in association with the distance pixel is acquired for each distance pixel from the distance image acquired by the above, and whether or not the acquired distance data exceeds a predetermined threshold value A determination unit that determines for each distance pixel; and a predetermined calculation for the pixel value of the imaging pixel corresponding to the distance pixel that is determined by the determination unit that the distance data exceeds the predetermined threshold among the captured images. And an image generating means for generating the output image by performing the above.

  In addition, the image processing method of the present invention performs a process for reducing the sharpness of a part or all of the captured image on the captured image obtained by capturing the imaging target, and outputs an output image with the reduced sharpness. A distance image obtained from a distance image sensor, wherein each of the distance pixels that constitute the distance image is associated with one of the imaging pixels that constitute the captured image. A distance image acquisition step of acquiring a distance image in which distance data indicating a distance from the distance image sensor of the imaging target is recorded in association with each distance pixel; and the distance image acquisition step From the acquired distance image, distance data recorded in association with the distance pixel is acquired for each distance pixel, and whether or not the acquired distance data exceeds a predetermined threshold value. A determination step for determining each distance pixel; and a predetermined calculation for the pixel value of the image pickup pixel corresponding to the distance pixel that is determined that the distance data exceeds the predetermined threshold in the determination step among the captured images. An image generation step of generating the output image by performing.

  Therefore, it is possible to perform a process of reducing (blurring) the sharpness of a part or all of an image with a simple configuration.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows an example of the data structure of picked-up image data. It is a figure which shows an example of the data structure of distance image data. It is a figure which shows an example of the data structure of output image data. It is a flowchart explaining the procedure which produces output image data, Comprising: The flow of the process which concerns on the procedure is shown. It is a flowchart which shows the flow of a pixel value calculation process. FIG. 7A is a diagram illustrating a configuration of a 3 × 3 blur filter, and FIGS. 7B to 7D are diagrams illustrating examples of filter coefficients given to each element as a matrix of the filter. . FIG. 8A is a diagram illustrating a configuration of a 5 × 5 blur filter, and FIGS. 8B to 8D are diagrams illustrating examples of filter coefficients given to each element as a matrix of the filter. . It is a figure which shows the structure of a 7 * 7 blurring filter. It is a block diagram which shows the structure of the image processing unit which concerns on other embodiment of this invention.

  Generally speaking, the image processing apparatus 101 according to the first embodiment and the image processing unit 1001 according to the second embodiment perform image processing that reduces (blurs) part or all of the sharpness of an image. It can be applied to various devices that process digital signals representing images. In particular, the image processing apparatus 101 and the image processing unit 1001 can be suitably applied to an apparatus that performs image processing on an image (captured) of a user captured by a videophone device.

  Therefore, in each embodiment, a case where the image processing apparatus 101 and the image processing unit 1001 are applied to an image processing apparatus included in a videophone apparatus will be described as an example. However, the application destination of the image processing apparatus 101 and the image processing unit 1001 is not limited to the videophone apparatus.

[Embodiment 1]
The image processing apparatus 101 according to the present embodiment will be described below with reference to FIGS.

[Schematic Configuration of Image Processing Apparatus 101]
FIG. 1 is a block diagram illustrating a schematic configuration of the image processing apparatus 101.

  The image processing apparatus 101 is a constituent element of the image transmission apparatus 100 for transmitting an image (captured) of a user in a videophone apparatus or the like to the other party's videophone apparatus as a call destination. The image transmission apparatus 100 includes a camera 106, a distance image sensor 107, and a transmission unit 108 in addition to the image processing apparatus 101.

  The image processing apparatus 101 includes a camera control unit 109, a sensor control unit 110, a main control unit 111, and an image transmission control unit (image transmission unit) 105.

  The camera 106 is a general camera represented by a digital camera, and takes an image of a user of a videophone device including the image processing device 101 and the background thereof. An object imaged by the camera 106 is also referred to as an “imaging object”. An image (hereinafter, also referred to as “captured image”) obtained by capturing an image with the camera 106 is output from the camera 106 as captured image data that can be subjected to digital image processing. The camera control unit 109 acquires captured image data output from the camera 106 and performs various controls on the camera 106.

  The distance image sensor 107 includes pixels (hereinafter referred to as “distance images”) in which distance data indicating the distance from the distance image sensor 107 itself to the captured object constitutes an image (hereinafter also referred to as “distance image”) obtained by the imaging. The pixel value is also recorded and output. As the distance image sensor 107, for example, a time-of-flight (TOF) type distance image sensor that obtains a distance by measuring a round-trip time until light hits an object and is reflected and returned can be used. In addition to the TOF type distance image sensor, as the distance image sensor 107, one having a function for obtaining a distance from a difference between images taken by two cameras can be used.

  Here, the distance image sensor 107 is provided so as to be able to take an image of the user and the background that the camera 106 is imaging. More specifically, between the distance image captured by the distance image sensor 107 and the captured image captured by the camera 106, the camera 106 and the A distance image sensor 107 is provided. Each of the pixels constituting the distance image captured by the distance image sensor 107 can be associated with any of the pixels (hereinafter also referred to as “imaging pixels”) that configure the captured image captured by the camera 106. It shall be.

  Hereinafter, the distance image data captured by the distance image sensor 107 is referred to as “distance image data”. The distance image data is output from the distance image sensor 107. The sensor control unit 110 acquires the distance image data output from the distance image sensor 107 and performs various controls on the distance image sensor 107.

  Here, in the distance image captured by the distance image sensor 107, the distance of the captured object from the distance image sensor 107 is recorded in units of pixels. The accuracy of distance and the unit of distance that can be measured differ depending on the individual sensor, but in this embodiment, the distance image sensor 107 that can output a pixel value corresponding to the distance from the distance image sensor 107 to the object. An example will be described.

  In order to make the positional relationship of each object in each captured image relatively similar in the camera 106 and the distance image sensor 107, the same camera is used as the camera 106 and the distance image sensor 107. Is preferred.

  On the other hand, as the camera 106 and the distance image sensor 107, different cameras may be used instead of the same camera. However, when each of the captured image data and the distance image data is acquired by using different cameras as the camera 106 and the distance image sensor 107, the two cameras need to be as close as possible. The reason for this is that if the two cameras are arranged apart from each other, a large shift in the positional relationship is likely to occur, that is, a pixel shift between the captured image data and the distance image data. This is because it tends to be large. When the angle of view and / or the size (positional relationship) of an object differ between the images captured by the two cameras, enlargement, reduction, trimming, or affine is performed on at least one of these images. These images are preferably converted to the same image size by performing conversion or the like.

  The main control unit 111 controls each unit of the camera 106, the distance image sensor 107, and the image transmission control unit 105. The main control unit 111 may be configured by a CPU (central processing unit) or hardware logic such as an ASIC (application specific integrate circuit). When the main control unit 111 is configured by a CPU, various programs stored in the memory are read out and executed in a primary storage unit configured by, for example, a RAM (random access memory) or the like.

  The main control unit 111 includes a pixel value acquisition unit 102, a distance image data analysis unit (distance image acquisition unit, determination unit) 104, and an image processing unit (image generation unit) 103.

  The pixel value acquisition unit 102 acquires captured image data from the camera control unit 109, and acquires a pixel value of each pixel constituting the captured image data for each pixel. Although details will be described later, since the captured image data is assumed to be an image captured by a general camera, the pixel value includes an R (red) component, a G (green) component, and a B (blue) component. Contains color information.

  The distance image data analysis unit 104 acquires the distance image data from the sensor control unit 110, and calculates the pixel value of each pixel constituting the distance image data, that is, the distance recorded in each distance pixel for each distance pixel. get. Then, the distance image data analysis unit 104 determines for each distance pixel whether or not the acquired distance exceeds a predetermined threshold (hereinafter referred to as “predetermined threshold T”).

  Here, the predetermined threshold T is a value for determining for each distance pixel whether the distance of the imaged object from the distance image sensor 107 is larger or smaller than the predetermined distance. A preferable value of the predetermined threshold T is a value indicating the distance from the distance image sensor 107 to the user. This is because, by setting a value indicating the distance from the distance image sensor 107 to the user as the predetermined threshold value T, an object in front of or next to the user is determined as an object to be shown to the other party (not blurred). This is because an object behind the user can be determined (blurred) as a background.

  First, the image processing unit 103 acquires, from the pixel value acquisition unit 102, the pixel value of each pixel constituting the captured image data for each pixel.

  Then, the image processing unit 103 performs pixel value calculation processing (predetermined calculation) described later on the pixel value of the imaging pixel corresponding to the distance pixel determined by the distance image data analysis unit 104 that the distance exceeds the predetermined threshold T. I do. The image processing unit 103 performs the process on each imaging pixel.

  As a result, the image processing unit 103 has a sharpness of an area in which an imaging target in which the distance from the distance image sensor 107 exceeds a predetermined distance represented by a predetermined threshold T among the captured images captured by the camera 106 is captured. An output image that is a blurred (blurred) image is generated.

  In the above description, the description of the series of processes in the pixel value acquisition unit 102, the image processing unit 103, and the distance image data analysis unit 104 is an explanation for one pixel constituting the captured image data. On the other hand, in the image processing apparatus 101, the pixel value acquisition unit 102, the image processing unit 103, and the distance image data analysis unit 104 perform the above-described series of processing for each of all the pixels constituting the captured image data. . Then, image data obtained by completing the above series of processing is output from the image processing unit 103 as output image data indicating an output image.

  The output image data output from the image processing unit 103 is supplied to the image transmission control unit 105. The image transmission control unit 105 outputs the supplied output image data to the transmission unit 108 outside the image processing apparatus 101.

  In general, in a videophone device, in order to transmit an image captured of a user to a videophone device that is a call destination, a mechanism that prevents synchronization of the sound and the image, and image data are compressed. Technology to do is necessary. These techniques are performed by the transmission unit 108. That is, the transmission unit 108 transmits the output image data supplied from the image transmission control unit 105 to the videophone device that is the call destination by a known transmission / reception method used in the videophone device.

[Data structure of captured image data, distance image data, and output image data]
Here, the captured image data, the distance image data, and the output image data will be described with reference to FIGS.

  FIG. 2 is a diagram illustrating an example of a data structure of captured image data. FIG. 2 shows captured image data having n pixels in the horizontal direction (X direction) and m pixels in the vertical direction (Y direction). Color information based on the RGB color system composed of R (red), G (green), and B (blue) components is included in each of the n × m pixels constituting the captured image data. It is recorded as a value.

  Hereinafter, in the captured image data, the R component recorded in the i-th pixel from the left and the j-th pixel from the top is denoted as Rij. Similarly, the G component and the B component recorded in the same pixel are denoted as Gij and Bij, respectively.

  FIG. 3 is a diagram illustrating an example of the data structure of the distance image data. FIG. 3 shows distance image data having n pixels in the horizontal direction (X direction) and m pixels in the vertical direction (Y direction). A distance data component (D) indicating a distance from the distance image sensor 107 to an object imaged by the distance image sensor 107 is recorded as a pixel value in each of the n × m pixels constituting the distance image data. ing.

  Hereinafter, in the distance image data, the component of the distance data recorded in the i-th pixel from the left and the j-th pixel from the top is denoted as Dij.

  Here, the captured image data and the distance image data are images obtained by capturing substantially the same position. The captured image data and the distance image data are both image data having n × m pixels. Therefore, it can be said that the captured image data and the distance image data are data obtained by imaging substantially the same imaging target with different imaging devices such as the camera 106 and the distance image sensor 107. Accordingly, the color information indicated by Rij, Gij, and Bij in the captured image data and the distance data indicated by Dij in the distance image data corresponding to the captured image data are captured at substantially the same position in the substantially identical imaging target. The information obtained is shown.

  FIG. 4 is a diagram illustrating an example of the data structure of the output image data. FIG. 4 shows output image data having n pixels in the horizontal direction (X direction) and m pixels in the vertical direction (Y direction). Each of n × m pixels constituting the output image data has color information based on the RGB color system composed of an R (red) component, a G (green) component, and a B (blue) component. It is recorded as a value.

  Hereinafter, in the output image data, the R component recorded in the i-th pixel from the left and the j-th pixel from the top is denoted as Roij. Similarly, the G component and the B component recorded in the same pixel are denoted by Goij and Boij, respectively.

  The output image data is created from the captured image data and the distance image data input to the image processing apparatus 101 according to the following procedure.

[Procedure for creating output image data]
A procedure (image processing method) for creating output image data will be described with reference to FIG.

  FIG. 5 is a flowchart for explaining a procedure for creating output image data, and shows a flow of processing of the image processing unit 103 and the like related to the procedure.

  As described above, the image processing for creating the output image data is performed for each pixel constituting the captured image data. Here, one pixel of the captured image data to be processed is referred to as a “target pixel”.

  Here, a description will be given by taking as an example a case where captured image data (see FIG. 2) having n pixels in the X direction and m pixels in the Y direction is input to the image processing apparatus 101. .

  In addition, here, the case where the number of pixels of the captured image data and the number of pixels of the distance image data are the same number (n × m) will be described as an example. Incidentally, when the number of pixels of the captured image data is different from the number of pixels of the distance image data, it is necessary to set the image of the distance image data to the same image size as the number of pixels of the captured image data in advance. As a method for adjusting the image size, a well-known technique such as enlargement, reduction, trimming, or affine transformation can be used.

  Further, here, for general description, a processing method of the i-th pixel from the left in the X direction coordinate and the j-th pixel from the top in the Y direction coordinate will be described.

  When the process is started (step ST51), the pixel value acquisition unit 102 acquires color information (pixel value) recorded in the target pixel from the captured image data acquired from the camera control unit 109. The color information recorded in the target pixel corresponds to the pixel value recorded in the target pixel, that is, the RGB components (Rij, Gij, Bij) (step ST52).

  The distance image data analysis unit 104 uses the distance image data acquired from the sensor control unit 110 to record the distance (from the distance image sensor 107 to the object imaged by the distance image sensor 107) related to the target pixel. Pixel value) is acquired as distance information. The distance related to the target pixel corresponds to the pixel value recorded in the target pixel, that is, the distance data Dij (distance image acquisition step ST53).

  In addition, the distance image data analysis unit 104 compares the pixel value (distance information) of the distance data Dij acquired in step ST53 with a predetermined distance threshold (predetermined threshold T) Th1 that is a parameter, and each of these is indicated. The magnitude of the distance is determined (determination step ST54).

  When the pixel value in the distance data Dij is equal to or smaller than the distance threshold Th1, it is determined that the distance from the distance image sensor 107 to the object imaged by the distance image sensor 107 is equal to or smaller than the predetermined distance indicated by the distance threshold Th1 with respect to the target pixel. . This determination result is that the object imaged by the distance image sensor 107 is closer to the distance image sensor 107 than the position corresponding to the predetermined distance or the object imaged by the distance image sensor 107 is related to the target pixel. It shows that it is in a position corresponding to the predetermined distance. When such a determination is made, the image processing unit 103 does not perform pixel value calculation processing described later on the color information recorded in the target pixel (step ST55).

That is, in step ST55, the image processing unit 103 records the color information acquired in step ST52 as it is for one pixel of the corresponding output image data, so that the coordinates (X, Y) of the output image data = Output pixel color information, which is the pixel value of the pixels constituting (i, j),
Roij = Rij
Goij = Gij
Boij = Bij
And In this case, the color information recorded in one pixel of the output image data corresponding to the target pixel is equal to the color information recorded in the target pixel.

  When the pixel value in the distance data Dij is larger than the distance threshold Th1, it is determined that the distance from the distance image sensor 107 to the object imaged by the distance image sensor 107 is greater than the predetermined distance indicated by the distance threshold Th1 for the target pixel. . This determination result indicates that the object captured by the distance image sensor 107 is farther from the distance image sensor 107 than the position corresponding to the predetermined distance with respect to the target pixel. When such a determination is made, the image processing unit 103 performs a pixel value calculation process described later on the color information recorded in the target pixel (image generation step ST56).

  In step ST <b> 56, the image processing unit 103 determines the pixel value recorded in the pixel of (X, Y) = (i, j) in the captured image data and the neighboring pixels, and distance data corresponding to each of them. Is used to calculate the output pixel color information. Details of the pixel value calculation processing will be described later.

  When the process shown in step ST55 or the process shown in step ST56 ends, the pixel value acquisition unit 102, the distance image data analysis unit 104, and the image processing unit 103 configure captured image data input to the image processing apparatus 101. A series of processes shown in steps ST52 to ST55 (or step ST56) is repeated for all the pixels.

  That is, when the above-described series of processing has not been performed for all of the pixels constituting the captured image data, the pixel value acquisition unit 102, the distance image data analysis unit 104, and the image processing unit 103 have not processed the target pixel. Then, the above-described series of processing is performed again.

  On the other hand, when the above-described series of processing has been performed for all the pixels constituting the captured image data, the image processing unit 103 outputs the obtained output image data to the image transmission control unit 105, and image processing is performed. Is terminated (step ST58).

[Pixel value calculation processing]
The pixel value calculation process will be described with reference to FIGS.

  FIG. 6 is a flowchart showing the flow of the pixel value calculation process.

  In the flowchart shown in FIG. 5, when the process proceeds to step ST56, a pixel value calculation process is started (step ST61), and the distance image data analysis unit 104 detects the pixel value (distance) of the distance data Dij acquired in step ST53. Information) is compared with a predetermined distance threshold (predetermined threshold T) Th2 which is a parameter different from the above-described distance threshold Th1, and the magnitude of the distance indicated by each of these is determined (step ST62).

  When the pixel value in the distance data Dij is equal to or smaller than the distance threshold Th2, the image processing unit 103 performs 3 × 3 blurring processing described later (step ST63).

  When the pixel value in the distance data Dij is larger than the distance threshold Th2, the distance image data analysis unit 104 sets the pixel value (distance information) of the distance data Dij acquired in Step ST53 as one of the above-described distance thresholds Th1 and Th2. Both are compared with a predetermined distance threshold (predetermined threshold) Th3, which is a different parameter, and the magnitude of the distance indicated by each of these is determined (step ST64).

  When the pixel value in the distance data Dij is equal to or smaller than the distance threshold Th3, the image processing unit 103 performs a 5 × 5 blurring process described later (step ST65).

  When the pixel value in the distance data Dij is larger than the distance threshold Th3, the image processing unit 103 performs a blurring process different from both the 3 × 3 blurring process according to step ST63 and the 5 × 5 blurring process according to step ST65 ( Step ST66).

  From here, each of the 3 × 3 blurring process and the 5 × 5 blurring process will be described. In the following, the R component recorded in the pixels constituting the coordinates (X, Y) in the captured image data is R [X] [Y], and the G component recorded in the pixels is G [X]. [Y] and the B component recorded in the same pixel are denoted as B [X] [Y], respectively.

  The 3 × 3 blurring process according to step ST63 is performed by the image processing unit 103 using a known blurring filter having filter coefficients corresponding to a matrix of 3 pixels × 3 pixels as shown in FIG. This can be done by using a "3x3 blur filter") and adjusting their respective filter coefficients.

  That is, the image processing unit 103 calculates the weighted average of the color information recorded in each of the target pixel and the neighboring pixels for each of the pixels constituting the captured image data, and calculates the R component, the G component, and the B component. Find for each of the ingredients. In addition, the image processing unit 103 replaces the color information recorded in the target pixel with the weighted average value obtained previously for each of the R component, the G component, and the B component.

  The image processing unit 103 calculates the pixel of interest having coordinates (X, Y) = (i, j) from the coefficient of the 3 × 3 blur filter,

Is calculated. Then, the image processing unit 103 performs R component Ro [i] [j] and G component Go [i] [j] in color information to be recorded in one pixel of the output image data corresponding to the target pixel. And Bo [i] [j], which is the B component,
Ro [i] [j] = Rnume / deno
Go [i] [j] = Gnume / deno
Bo [i] [j] = Bnume / deno
Ask more.

  Then, the image processing unit 103 converts the color information (R [i] [j], G [i] [j], and B [i] [j]) recorded in the target pixel in the captured image data, respectively. And the calculated weighted average (Ro [i] [j], Go [i] [j], and Bo [i] [j]).

  Each element (as a matrix) in the 3 × 3 blur filter shown in FIG. 7A describes a filter coefficient called Kpq for the p-th element from the left and the q-th element from the top. In particular, the element describing the filter coefficient K22 is an element corresponding to the target pixel, and the other elements are elements corresponding to the eight pixels around (near) the target pixel.

  For example, a value pattern as shown in FIG. 7B, FIG. 7C, or FIG. 7D can be used as the filter coefficient given to each element.

  In FIG. 7B, since all the filter coefficients Kpq have the same value (1), a table for storing a plurality of filter coefficients as elements thereof becomes unnecessary.

  In FIG. 7C, two elements that are symmetrical to each other with respect to the element of the filter coefficient K22 (4) in the 3 × 3 blurring filter have the same filter coefficient Kpq value (1 or 2). . In FIG. 7C, the filter coefficient Kpq has a larger value as the element is closer to the filter coefficient K22. For this reason, the 3 × 3 blur filter having each filter coefficient Kpq shown in FIG. 7C has characteristics as a light LPF (low-pass filter). Further, since the sum of the filter coefficients Kpq is a power of 2, it is possible to mount each member constituting the image processing apparatus 101 so that the color information of the output image data is calculated not by division but by shift calculation. Become.

  In FIG. 7D, each filter coefficient Kpq is represented by only 0 or 1, and, as in FIG. 7C, the total of each filter coefficient Kpq is a power of 2. Is a feature.

  The 5 × 5 blurring process according to step ST65 is performed by the image processing unit 103 with a blurring filter having filter coefficients corresponding to a well-known matrix of 5 pixels × 5 pixels as shown in FIG. Can be implemented by using a "5x5 blur filter") and adjusting their respective filter coefficients.

  Each element (as a matrix) in the 5 × 5 blur filter shown in FIG. 8A describes a filter coefficient Lpq for the p-th element from the left and the q-th element from the top.

  In the 5 × 5 blur filter as the image processing unit 103, first, the element in which the filter coefficient L33 is described corresponds to the target pixel, as in the case of the 3 × 3 blur process. The sum of the values of the filter coefficients Lpq is set as the denominator (deno5). Further, for each component of RGB, a weighted sum is calculated for each element of the 5 × 5 blur filter by the value of the filter coefficient Lpq of the corresponding element and the color information (pixel value) recorded in the element. , Rnume5, Gnume5, and Bnume5, respectively. Rnume5, Gnume5, and Bnume5 are molecules for deno5, respectively.

Then, the image processing unit 103 performs R component Ro [i] [j] and G component Go [i] [j] in color information to be recorded in one pixel of the output image data corresponding to the target pixel. And Bo [i] [j], which is the B component,
Ro [i] [j] = Rnume5 / deno5
Go [i] [j] = Gnume5 / deno5
Bo [i] [j] = Bnume5 / deno5
Ask more.

  For example, a value pattern as shown in FIG. 8B, FIG. 8C, or FIG. 8D can be used as the filter coefficient given to each element.

  In FIG. 8B, since all the filter coefficients Lpq have the same value (1), a table for storing a plurality of filter coefficients as elements thereof becomes unnecessary.

  In FIG. 8C, two elements that are symmetrical to each other with respect to the element of the filter coefficient L33 (4) in the 5 × 5 blur filter are represented by the same value (0, 1, or 2) of the filter coefficient Lpq. ). Also, in FIG. 8C, the filter coefficient Lpq has a larger value as the element is closer to the filter coefficient L33. For this reason, the 5 × 5 blurring filter having each filter coefficient Lpq shown in FIG. 8C has characteristics as a light LPF (low-pass filter). Furthermore, since the sum of the filter coefficients Lpq is a power of 2, it is possible to mount each member constituting the image processing apparatus 101 so that the color information of the output image data is calculated not by division but by shift calculation. Become.

  In FIG. 8D, each filter coefficient Lpq is represented by only 0 or 1, and, as in FIG. 8C, the total of each filter coefficient Lpq is a power of 2. Is a feature.

  The blur processing according to step ST66 is performed by the image processing unit 103 using a known blur filter having filter coefficients corresponding to a matrix of 7 pixels × 7 pixels as shown in FIG. 9 (hereinafter, “7 × 7 blur filter”). Can be implemented.

  Note that each element (as a matrix) in the 7 × 7 blur filter shown in FIG. 9 describes a filter coefficient Mpq for the p-th element from the left and the q-th element from the top.

  In the 7 × 7 blur filter as the image processing unit 103, as in the case of the 5 × 5 blur process, first, an element in which the filter coefficient M44 is described is associated with the target pixel. The total sum of the values of the filter coefficients Mpq is set as the denominator (deno7). For each component of RGB, a weighted sum is calculated for each element of the 7 × 7 blur filter by the value of the filter coefficient Mpq of the corresponding element and the color information (pixel value) recorded in the element. , Rnume7, Gnume7, and Bnume7, respectively. Rnume7, Gnume7, and Bnume7 are molecules for deno7, respectively. That is,

It becomes. In these mathematical formulas, x is p in Mpq, and y is q in Mpq.

Then, the image processing unit 103 performs R component Ro [i] [j] and G component Go [i] [j] in color information to be recorded in one pixel of the output image data corresponding to the target pixel. And Bo [i] [j], which is the B component,
Ro [i] [j] = Rnume7 / deno7
Go [i] [j] = Gnume7 / deno7
Bo [i] [j] = Bnume7 / deno7
Ask more.

  Note that, in each blurring process according to step ST63, step ST65, and step ST66, when the pixel of interest is closer to the end of the captured image data and there is no pixel to be multiplied by the filter coefficient, all of the RGB components are 0. The processing may be continued assuming that a certain pixel exists.

  With the above method, the image processing unit 103 can obtain the color information of the output image data by the pixel value calculation process.

  In the present embodiment, a 3 × 3 blurring process is used for the 3 × 3 blurring process according to step ST63, and a 5 × 5 blurring filter is used for the 5 × 5 blurring process according to step ST65. For the 7 × 7 blurring process according to step ST66, a 7 × 7 blurring filter was used. However, each of the blur processing and the blur filter is merely an example, and the size of the blur filter and the detailed procedure of the blur processing are not particularly limited.

Furthermore, in this embodiment,
Distance threshold Th1 <Distance threshold Th2 <Distance threshold Th3
Each of the distance thresholds Th1 to Th3 is set so that Also, a filter coefficient having a low LPF characteristic is set for the 3 × 3 blurring process, a filter coefficient having a high LPF characteristic is set for the 5 × 5 blurring process, and the blurring process according to step ST66 is used for the 5 × 5 blurring process. A filter coefficient having an LPF characteristic higher than the determined coefficient is set. Thereby, the image processing apparatus 101 can realize blurring processing according to the distance.

  As a result, the videophone apparatus provided with the image processing apparatus 101 can clearly show the other party what he / she wants to show to the other party who is talking at a short distance, while being far away from the background. It is possible to blur an object that the user does not want to show to the other party, create an image that is not shown to the other party, and send the image to the other party's videophone device.

  And in order to implement | achieve the image processing apparatus 101, the image process part 103 is the pixel value of the attention pixel which is 1 pixel in the captured image data used as the object which performs pixel value calculation processing, and this attention pixel in this captured image data The pixel value calculation processing is performed using the pixel value of each pixel adjacent to the pixel.

  In addition, as the pixel value calculation process, the image processing unit 103 changes the pixel value of the target pixel to the weighted average value of the pixel value of the target pixel and the pixel value of each pixel adjacent to the target pixel in the captured image data. It is the structure to do.

  Further, the image processing unit 103 records the filter coefficient corresponding to the distance data Dij recorded in association with one pixel constituting the distance image data corresponding to the target pixel, the pixel value of the target pixel, and the imaging After multiplying the pixel value of each pixel adjacent to the target pixel in the image data, the weighted average value is calculated, and the pixel value of the target pixel is changed to the weighted average value.

  As a result, image processing according to the distance from the distance image sensor 107 to the object imaged by the distance image sensor 107, such as blurring the object farther from the distance image sensor 107, is performed on the image captured by the camera 106. It becomes possible.

  That is, since the pixel value calculation process described above can be realized by the blurring process of the captured image using a known blur filter (low-pass filter), the state before the blur process is performed on the area subjected to the pixel value calculation process. Can be generated (leaving the atmosphere as a background).

  Further, by changing the filter coefficient of the blur filter according to the distance data, in the captured image, the degree of blur is reduced between an object at a distance close to the distance image sensor 107 and an object at a distance far from the distance image sensor 107. Can be changed. Accordingly, it is possible to perform processing such as blurring an image of an object at a distance far from the distance image sensor 107 more significantly than an image of an object at a distance close to the distance image sensor 107.

  Further, since the image processing apparatus 101 can transmit the output image to the outside by the configuration including the image transmission control unit 105, the external apparatus can use the output image. For example, in the case where the image processing apparatus 101 according to the present embodiment is provided in the videophone apparatus as a component of the videophone apparatus, the videophone apparatus outputs an output image to the videophone apparatus that is the call destination. It becomes possible to transmit.

[Embodiment 2]
The image processing unit 1001 according to the present embodiment will be described below with reference to FIG.

  FIG. 10 is a block diagram showing the configuration of the image processing unit 1001.

  The image processing unit 1001 is a component of the image transmission device 1000 for transmitting an image (captured) of a user in a videophone device to a videophone device that is a call destination. The image transmission apparatus 1000 includes a camera 1006, a camera 1007, and a transmission unit 1008 in addition to the image processing unit 1001.

  The image processing unit 1001 includes an image processing device 1002, an image processing device 1003, a distance image calculation unit (first distance calculation unit) 1004, and a distance image calculation unit (second distance calculation unit) 1005.

  The camera 1006 has the same configuration as the camera 106 (see FIG. 1).

  The camera 1007 has the same configuration as the camera 1006 and thus the camera 106, and is arranged at a predetermined interval from the camera 1006.

  An image (captured image) captured by the camera 1006 is output from the camera 1006 as captured image data that can be subjected to digital image processing. Similarly, an image (captured image) captured by the camera 1007 is output from the camera 1007 as captured image data that can be subjected to digital image processing.

  The distance image calculation unit 1004 acquires captured image data output from the camera 1006 and captured image data output from the camera 1007. In addition, the distance image calculation unit 1004 is configured to be able to refer to the interval between the camera 1006 and the camera 1007 in advance.

  The distance image calculation unit 1004 is based on the positional deviation of each captured object in the captured image data output from the camera 1007 with respect to the captured image data output from the camera 1006 and the interval between the camera 1006 and the camera 1007. The distance from the camera 1006 to the object imaged by the camera 1006 is obtained for each pixel constituting the captured image data output from the camera 1006 by trigonometry.

  Then, the distance image calculation unit 1004 corresponds to the image data in which the pixels are arranged in association with the pixels constituting the captured image data output from the camera 1006, each of the pixel values indicating the distance obtained above. The distance image data is generated by recording in the pixel, and the distance image data is output to the image processing apparatus 1002.

  In addition, the distance image calculation unit 1005 acquires captured image data output from the camera 1006 and captured image data output from the camera 1007. Further, the distance image calculation unit 1005 has a configuration in which the distance between the camera 1006 and the camera 1007 can be referred to in advance.

  The distance image calculation unit 1005 is based on the positional deviation of each captured object in the captured image data output from the camera 1007 with respect to the captured image data output from the camera 1006 and the interval between the camera 1006 and the camera 1007. The distance from the camera 1007 to the object imaged by the camera 1007 is obtained for each pixel constituting the captured image data output from the camera 1007 by trigonometry.

  Then, the distance image calculation unit 1005 corresponds to the image data in which each pixel value indicating the distance obtained above is associated with the pixel constituting the captured image data output from the camera 1007 and the pixel is arranged. The distance image data is created by recording in the pixel, and the distance image data is output to the image processing apparatus 1003.

  Both the image processing apparatuses 1002 and 1003 have the same configuration as the image processing apparatus 101 (see FIG. 1).

  The image processing apparatus 1002 acquires the captured image data output from the camera 1006 and the distance image data output from the distance image calculation unit 1004, and performs the same processing as that of the image processing apparatus 101. Then, the image processing device 1002 outputs the output image data obtained by the processing to the transmission unit 1008.

  The image processing apparatus 1003 acquires the captured image data output from the camera 1007 and the distance image data output from the distance image calculation unit 1005, and performs the same processing as that of the image processing apparatus 101. Then, the image processing device 1003 outputs the output image data obtained by the processing to the transmission unit 1008.

  The transmission unit 1008 transmits the two output image data supplied from each of the image processing apparatuses 1002 and 1003 to a videophone apparatus that is a call destination by a known transmission / reception method used in the videophone apparatus.

  Note that the image processing unit 1001 is an example for a three-dimensional videophone using two cameras 1006 and 1007. A three-dimensional image using the parallax of the right eye and the left eye is characterized by facilitating the understanding of the object that matches the parallax in order to blur the image at a distance and clearly show what is in the foreground.

[Additional Notes]
Finally, the functions of the main control unit 111 of the image processing apparatuses 101, 1002, and 1003 may be configured by hardware logic, or may be realized by software using a CPU as follows.

  When implemented by software, the main control unit 111 includes a CPU that executes instructions of a control program that implements each function, a ROM (read only memory) that stores the program, a RAM that expands the program, the program, and various types of programs. A storage device (recording medium) such as a memory for storing data is provided. An object of the present invention is to enable a computer to read program codes (execution format program, intermediate code program, source program) of a control program for the image processing apparatus 101 and the image processing unit 1001 that are software that realizes the above-described functions. This can also be achieved by supplying the recorded recording medium to the image processing apparatus 101 and the image processing unit 1001 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. IC cards (including memory cards) / optical cards, semiconductor memories such as mask ROM / EPROM / EEPROM / flash ROM, logic circuits such as PLD (Programmable Logic Device), etc. it can.

  Further, the image processing apparatus 101 and the image processing unit 1001 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Further, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), IEEE802.11 radio, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance), mobile phone network, satellite line, terrestrial digital network, etc. .

  Thus, in this specification, the means does not necessarily mean physical means, but includes cases where the functions of the means are realized by software. Further, the function of one means may be realized by two or more physical means, or the functions of two or more means may be realized by one physical means.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  The present invention can be applied to an apparatus for processing a digital signal representing an image. In particular, the present invention can be suitably applied to a videophone device that transmits an image captured by a user or the like to a call destination. When applied to a videophone device, the present invention reliably shows an image that the user wants to show to the other party, deletes the image that the user does not want to show to the other party to such an extent that the situation at the call site can be conveyed, and the circuit scale There is an effect that it is possible to suppress an increase in the amount of.

101 Image Processing Device 102 Pixel Value Acquisition Unit 103 Image Processing Unit (Image Generation Unit)
104 Distance image data analysis unit (distance image acquisition means, determination means)
105 Image transmission control unit (image transmission means)
106 Camera 107 Distance Image Sensor 1001 Image Processing Units 1002 and 1003 Image Processing Device 1004 Distance Image Calculation Unit (First Distance Calculation Unit)
1005 Distance image calculation unit (second distance calculation means)
1006 and 1007 cameras

Claims (8)

An image processing device that performs processing for reducing the sharpness of a part or all of the captured image on a captured image obtained by capturing an imaging target, and outputs an output image with the reduced sharpness,
A distance image obtained from a distance image sensor, each distance pixel being a pixel constituting the distance image is associated with one of imaging pixels being pixels constituting the captured image, and Distance image acquisition means for acquiring a distance image in which distance data indicating a distance from the distance image sensor is recorded in association with each distance pixel;
From the distance image acquired by the distance image acquisition means, the distance data recorded in association with the distance pixel is acquired for each distance pixel, and whether the acquired distance data exceeds a predetermined threshold Determination means for determining each pixel;
Image generation for generating the output image by performing a predetermined calculation on the pixel value of the imaging pixel corresponding to the distance pixel determined by the determination unit that the distance data exceeds the predetermined threshold among the captured images And an image processing apparatus.   The image generation means performs the predetermined calculation using a pixel value of the imaging pixel to be subjected to the predetermined calculation and a pixel value of the imaging pixel adjacent to the imaging pixel. The image processing apparatus described. The image generation means includes
As the predetermined calculation, the pixel value of the imaging pixel to be subjected to the predetermined calculation is changed to a weighted average value of the pixel value of the imaging pixel and the pixel value of the imaging pixel adjacent to the imaging pixel. The image processing apparatus according to claim 2. The image generation means includes
The coefficient corresponding to the distance data recorded in association with the distance pixel corresponding to the imaging pixel to be subjected to the predetermined calculation is set to the pixel value of the imaging pixel and the pixel value of the imaging pixel adjacent to the imaging pixel. The image processing apparatus according to claim 3, wherein the weighted average value is calculated after multiplication.   The image processing apparatus according to claim 1, further comprising an image transmission unit that transmits the output image generated by the image generation unit to the outside of the image processing apparatus. An image processing method for performing processing for reducing the sharpness of a part or all of the captured image on a captured image obtained by capturing an imaging target, and outputting an output image with the reduced sharpness,
A distance image obtained from a distance image sensor, each distance pixel being a pixel constituting the distance image is associated with one of imaging pixels being pixels constituting the captured image, and A distance image acquisition step for acquiring a distance image in which distance data indicating a distance from the distance image sensor is recorded in association with each distance pixel;
From the distance image acquired in the distance image acquisition step, distance data recorded in association with the distance pixel is acquired for each distance pixel, and whether or not the acquired distance data exceeds a predetermined threshold A determination step for determining each distance pixel;
Among the captured images, an image that generates the output image by performing a predetermined calculation on the pixel value of the captured pixel corresponding to the distance pixel that has been determined that the distance data exceeds the predetermined threshold in the determination step. And a generation step.   A control program for operating a computer included in the image processing apparatus according to any one of claims 1 to 5, wherein the control program causes the computer to function as each of the means.   A computer-readable recording medium on which the control program according to claim 7 is recorded.

Images