JP2016085579A - Image processing apparatus and method for interactive device, and the interactive device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform correction so that visual lines of interlocutors match each other, and also solve the "occlusion" problem and achieve face beautification at the same time, in an image processing apparatus for an interactive device.SOLUTION: An image processing apparatus for an interactive device includes a first dictionary memory for storing a first dictionary matrix (Φ) for a face reconfiguration process for a blank filling process that occurs due to visual line correction and a second dictionary memory for storing a second dictionary matrix (Ψ) for a face beautification process. When an object function is minimized that includes a first sparse code vector (α) for the first dictionary matrix (Φ), a second sparse code vector (β) for the second dictionary matrix (Ψ), and linear conversion (L) from reconfigured face image data to face beautified simple image data, the optimum value of each of the first sparse code vector (α), the second sparse code vector (β), and the linear conversion (L) is obtained on the basis of a face image vector (x), and face image data that has undergone the face reconfiguration process and face beautification process is generated, and the generated face image data are output.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus and method for an interactive apparatus used for a videophone or a video conference, and an interactive apparatus.

  With the advent of imaging technology and network technology that enables low-cost, high-quality video capture and highly reliable broadband transmission, video conferencing systems that connect two people physically separated by long distances, for example, It is a ubiquitous system that can be easily implemented using tools such as Skype and Google Plus Hangouts.

  A common problem with these tools is the problem of gaze mismatch. The capture camera is typically placed on the upper or lower side of the display monitor, but since the user talks while looking at the face of the other party, the users cannot talk with each other with the same line of sight. There was a problem of reducing the quality of communication.

  In taking advantage of recent advances in 3D imaging (eg, see Non-Patent Document 1), one common approach to the problem of gaze mismatch is that a texture image and a viewpoint image with depth capture camera (eg, (See Non-Patent Documents 3 to 5), a screen (virtual image) using rendering based on depth images (DEPth-Image-Based Rendering (hereinafter referred to as DIBR)). This is to synthesize a viewpoint image viewed from the center (for example, see Non-Patent Documents 6 to 8).

  Further, in Patent Document 1, in order to provide an interactive apparatus that can easily realize a natural interactive environment in which the lines of sight of the interlocutors match in interactive communication, an image processing module includes a person included in input image data. The positions of the iris and pupil are changed by changing the pixels of the eyelid area (exposed area of the eyeball) of the person included in the image data so that the line of sight of the image faces the front of the image data. Here, the image processing module includes a face recognition unit, a line-of-sight determination unit, and a line-of-sight change unit. The face recognition unit extracts information on a vector (face vector) that defines the orientation of the entire face based on the result of the face recognition process. When it is determined that the face is facing the display screen based on the orientation of the face vector, the line-of-sight changing unit changes the pixels in the eyelid region so that the person's line of sight faces the front of the image data.

  Further, Patent Documents 2 and 3 disclose a method of correcting the line of sight of a caller or the like by matching them.

JP 2009-246408 A JP 2005-117106 A JP 2013-182616 A JP 2004-147288 A

M. Tanimoto, M. P. Tehrani, T. Fujii and T. Yendo, "Free-Viewpoint TV," IEEE Signal Processing Magazine, vol.28, no.1, January 2011. D. Tian, P.-L. Lai, P. Lopez and C. Gomila, "View Synthesis Techniques for 3D Video," Applications of Digital Image Processing XXXII, Proceedings of the SPIE, 7443 (2009), 2009. S.-B. Lee, I.-Y. Shin and Y.-S. Ho, "Gaze-corrected View Generation Using Stereo Camera System for Immersive Videoconferencing," IEEE Transactions on Consumer Electronics, vol.57, no.3, pp.1033-1040, August 2011. C. Kuster, T. Popa, J.-C. Bazin, C. Gotsman and M. Gross, "Gaze Correction for Home Video Conferencing," ACM SIG-GRAPH Asia, vol.31, no.6, November 2012. W. Eng, D. Min, V.-A.Nguyen, J. Lu and M. Do, "Gaze Correction for 3D Tele-immersive Communication System," 11th IEEE IVMSP Workshop: 3D Image / Video Technologies and Applications, Seoul, Korea, June 2013. K.-J.Oh, S. Yea and Y.-S.Ho, "Hole-Filling Method Using Depth Based In-Painting Fore View Synthesis in Free Viewpoint Television FTV and 3D Video," Picture Coding Symposium, Chicago, IL, May 2009. S. Reel, G. Cheung, P. Wong and L. Dooley, "Joint Texture-Depth Pixel Inpainting of Dis-occlusion Holes in Virtual View Synthesis," APSIPA ASC, Kaohsiung, Taiwan, October 2013. B. Macchiavello, C. Dorea, EM Hung and G. Cheung and I. Bajic, "Low-Saliency Prior for Disocclusion Hole Filling in DIBR-Synthesized Images," IEEE International Conference on Acoustics, Speech and Signal Processing, Florence, Italy, May 2014. T. Leyvand, D. Cohen-Or, G. Dror and D. Lischinski, "Data-Driven Enhancement of Facial Attractiveness," ACM SIG-GRAPH, vol.27, no.3, August 2008. H. Yue, X. Sun, J. Yang and F. Wu, "Cloud-based Image Coding for Mobile Devices-Towards Thousands To One Compression," IEEE Transactions on Multimedia, vol.15, no.4, pp.845- 857, June 2013. J. Gemmell, k. Toyama, CL Zitnick, T. Kang, and S. Seitz, "Gaze awareness for video conferencing: A software approach, IEEE Multimedia, vol.7, no.4, pp. 2635, Oct-Dec 2000 . A. Criminisi, J. Shotton, A. Blake, and P. Toor, "Gaze manipulation for one-to-one teleconferencing, IEEE International Conference on Computer Vision, Nice, France, October 2003. R. Yang and Z. Zhang, "Eye gaze correction with stereovision for video-teleconferencing, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.26, no.7, pp.956960, July 2004. G. Cheung, J. Ishida, A. Kubota, A. Ortega, "Transform Domain Sparsification of Depth Maps using Iterative Quadratic Programming," IEEE International Conference on Image Processing, Brussels, Belgium, September 2011. W. Hu, G. Cheung, X. Li, O. Au, "DepthMap Compression using Multi-resolution Graph-based Transform for Depth-image based Rendering," IEEE International Conference on Image Processing, Orlando, FL, September 2012. Yael Eisenthal, Gideon Dror, Eytan Ruppin, "Facial Attractiveness: Beauty and the Machine," Neural Computation vol.18, no.1, pp.119-142, 2006. M. Aharon, M. Elad, and AM Bruckstein, "The K-SVD: An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation," IEEE Transactions On Signal Processing, vol.54, no.11, pp.4311-4322, November 2006. X. Liu, D. Zhai, D. Zhao, W. Gao, "Image Super-Resolution via Hierarchical and Collaborative Sparse Representation," Proceeding of Data Compression Conference, pp.93-102, 2013. W. Dong, L. Zhang, G. Shi, and X.Wu ,, "Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization," IEEE Transactions On Image Processing, vol.20, no.7, pp 1838-1857, July 2011. Allen Y. Yang, Z. Zhou, A. Ganesh, S. Sastry, and Y.Ma, "Fast 11-minimization algorithms for robust face recognition," IEEE Transactions On Image Processing, vol.22, no.8, pp. 3234-3246, Aug. 2013. X. Zhu and D. Ramanan, "Face detection, pose estimation, and landmark localization in the wild," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2879-2886, June 2012. I. Ahn and C. Kim, "Depth-based disocclusion filling for virtual view synthesis," in IEEE International Conference on Multimedia and Expo, Melbourne, Australia, July 2012. Tommer Leyvand, Daniel Cohen-Or, Gideon Dror and Dani Lischinski, "Data-driven enhancement of facial attractiveness," in ACM SIG-GRAPH, vol. Vol.27, no.3, August 2008. M. Dumont, S. Maesen, S. Rogmans, and P. Bekaert, "A prototype for practical eye-gaze corrected video chat on graphics hardware," in Inter-national Conference on Signal Processing and Multimedia Applications, Porto, Portugal, July 2008. L. Wolf, Z. Freund, and S. Avidan, "An eye for an eye: A single camera gaze-replacement method," in 11th European Conference on Computer Vision, Crete, Greece, September 2010. S.-B. Lee and Y.-S. Ho, "Generation of eye contact image using depth camera for realistic telepresence," in APSIPA ASC, Kaohsiung, Taiwan, October 2013. D. Guo and T. Sim, "Digital face makeup by example," in IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, June 2007. C.-Y. Yang, S. Liu, and M.-H. Yang, "Structured face hallucination," in IEEE International Conference on Computer Vision and Pattern Recognition, Portland, OR, June 2013. J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, T. Cog, J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, quipes-projets Willow, and EN Suprieure, "Supervised dictionary learning," 2008. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online dictionary learning for sparse coding," in Proceedings of the 26th Annual International Conference on Machine Learning, ser.ICML '09. New York, NY, USA : ACM, 2009, pp. 689-696. J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang, "Coupled dictionary training for image super-resolution," Image Processing, IEEE Transactions on, vol. 21, no. 8, pp. 3467 -3478, Aug 2012. S. Wang, D. Zhang, Y. Liang, and Q. Pan, "Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis," in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, June 2012, pp. 2216-2223. J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, "Robust face recognition via sparse representation," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 31, no. 2, pp. 210-227, Feb 2009. D. Lowe, "Object recognition from local scale-invariant features," in IEEE International Conference on Computer Vision, September 1999. Y. Yang, B. Geng, Y. Cai, A. Hanjalic, and X.-S. Hua, "Object retrieval using visual query context," in IEEE Transactions on Multimedia, vol. 13, no. 6, December 2011, pp. 1295-1307. I. Daubechies, R. Devore, M. Fornasier, and S. Gunturk, "Iteratively reweighted least squares minimization for sparse recovery," in Commun. Pure Appl. Math, 2009. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, 2004. M. Taylor and C. Creelman, "PEST: Efficient estimates on probability functions," in The Journal of the Acoustical Society of America, vol. 41, 1988, pp. 782-787. H. Hadizadeh, I. Bajic, and G. Cheung, "Video error concealment using a computation-efficient low saliency prior," in IEEE Transactions on Multimedia, vol. 15, no.8, December 2013, pp. 2099-2113. D. J. Sheskins, Handbook of Parametric and Nonparametric Statistical Procedures. Chapman & Hall / CRC, 2007.

  However, an important problem of the DIBR approach described above is how to fill the occlusion area in order to solve the so-called “occlusion” problem in which occlusion areas occur when the subject textures overlap. This is the “shield removal hole filling problem”. The spatial region in the virtual image shielded by the foreground components at the shooting viewpoint with the digital camera contains pixel defects and needs to be completed to satisfy. Filling for images synthesized with DIBR has been dealt with in Non-Patent Documents 2 to 4, but as far as the inventor knows, no algorithm is yet to fill in “occlusion” for general human faces. Has not yet been proposed.

  The object of the present invention is to solve the above-mentioned problems, and in an image processing apparatus for an interactive apparatus, corrects the line of sight of the conversation person to coincide with each other, simultaneously solves the problem of “occlusion” and also performs facial beautification An image processing apparatus and method for an interactive apparatus, and an interactive apparatus.

An image processing apparatus according to a first invention
A first dictionary configured by clustering after extracting a plurality of patches from a plurality of face image data, and storing a first dictionary matrix (Φ) for face reconstruction processing for hole filling processing generated by line-of-sight correction Memory,
A second dictionary memory configured by clustering a plurality of components of a beautiful face from a plurality of face image data and storing a second dictionary matrix (Ψ) for face beautification processing;
Based on the input face image vector (x), a first sparse code vector (α) for the first dictionary matrix (Φ) and a second sparse code vector for the second dictionary matrix (Ψ) (Β) and a linear transformation matrix (L) from the reconstructed face image data to the simple image data subjected to the face beautification, for the face reconstruction and the face beautification By obtaining respective optimum values of the first sparse code vector (α), the second sparse code vector (β) and the linear transformation matrix (L) when minimizing the objective function of And image processing means for generating and outputting face image data obtained by performing face reconstruction processing and face beautification processing on the face image vector (x).

In the image processing apparatus, the objective function is a warping operator W that warps input face image data (A) to face-beautified face image data (B) instead of the linear transformation matrix (L). (A, B) and a mapping function S (•) for mapping a pixel region vector to a vector in a selected feature space using a scale invariant feature transformation method,
The image processing means respectively optimizes the first sparse code vector (α) and the second sparse code vector (β) when the objective function for the face reconstruction and the beautification is minimized. By obtaining a value, face image data obtained by performing face reconstruction processing and face beautification processing on the input face image vector (x) is generated and output.

In the image processing apparatus, the objective function is expressed by the following equation:
Here, μ is a predetermined constant, u is a weight vector for modulating the second sparse code vector (β) to satisfy a predetermined linear constraint,
The image processing means obtains the first sparse code vector (α) and the second sparse code vector (β) using the above formula, and performs face reconstruction on the input face image vector (x). It is characterized by generating and outputting face image data that has undergone configuration processing and face beautification processing.

  Further, in the image processing device, provided in the preceding stage of the image processing means, the line of sight of the face image data to be captured is corrected so that the line of sight of the face image data matches the line of sight of the conversation person. The image processing apparatus further includes line-of-sight correction processing means for outputting the face image vector (x).

  Furthermore, in the image processing device, provided after the image processing means, the face image data from the image processing means is adjusted by adjusting the face contour using a plurality of face contour data. The image processing apparatus further comprises a face contour adjusting means for outputting the face image data.

  Still further, in the image processing apparatus, a warping process is provided for the face image data from the face contour adjusting unit, which is provided after the face contour adjusting unit, so as to include the background of the captured face image data. The image processing apparatus further includes image warping processing means for performing and outputting face image data.

The dialogue apparatus according to the second invention is
Transmitting / receiving means for receiving face image data from the other party's dialogue device and transmitting the face image data of the dialogue device to the other party's dialogue device;
An interactive device comprising a display unit for displaying each face image data,
The image processing apparatus according to any one of claims 1 to 8, comprising:
The transmission / reception means transmits the face image data from the image processing apparatus to the other party's dialogue apparatus.

An image processing method according to a third invention is:
A first dictionary configured by clustering after extracting a plurality of patches from a plurality of face image data, and storing a first dictionary matrix (Φ) for face reconstruction processing for hole filling processing generated by line-of-sight correction Memory,
An image processing apparatus comprising: a second dictionary memory configured by clustering a plurality of components of a beautiful face from a plurality of face image data and storing a second dictionary matrix (Ψ) for face beautification processing An image processing method for
Based on the input face image vector (x), the image processing apparatus performs the first sparse code vector (α) for the first dictionary matrix (Φ) and the second dictionary matrix (Ψ). An objective function comprising a second sparse code vector (β) and a linear transformation (L) from the reconstructed face image data to the simplified face data subjected to face beautification, wherein the face reconstruction and Obtaining respective optimum values of the first sparse code vector (α), the second sparse code vector (β), and the linear transformation (L) when minimizing the objective function for the beautification of the face The method includes generating and outputting face image data obtained by performing face reconstruction processing and face beautification processing on the input face image vector (x).

  According to the present invention, in an image processing apparatus for an interactive apparatus, correction can be performed so that the lines of sight of the interlocutor coincide with each other, the problem of “occlusion” can be solved simultaneously and face beautification can be performed.

It is a block diagram which shows the structure of the dialogue apparatus provided with the image processing part 15 which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the dialogue apparatus provided with 15 A of image processing parts which concern on Embodiment 2 of this invention. It is a figure for demonstrating the search of the dual code vector performed in 15 A of image processing parts of FIG. It is a figure which shows an example of the landmark used by the alignment (position alignment) of the typical image performed in 15 A of image processing parts of FIG. FIG. 2 shows a result of image processing by the image processing unit 15A in FIG. 2 and is a photograph of the image processing result at a small imaging angle with respect to an expressionless frame. It is an image processing result of composition and face beautification processing. FIG. 2 is an image processing result by the image processing unit 15A in FIG. 2 and is a photograph of the image processing result at a small imaging angle with respect to a frame with a slightly smiling expression, the upper row is an image processing result of only the hole filling processing, and the lower row is an image processing result. It is an image processing result of image reconstruction and face beautification processing. It is a table which shows the vote result with respect to the front image by which the image processing part 15A of FIG. FIG. 2 shows the result of image processing by the image processing unit 15A in FIG. 2, which is a result of integrating the face beautified face image into the captured image, the upper row is the original captured face image, and the lower row is the integrated image. It is a face image.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In addition, in each following embodiment, the same code | symbol is attached | subjected about the same component.

Embodiment 1. FIG.
1-1. Preface.
In the present embodiment, it is proposed to solve the masking removal hole filling problem for human faces together with the face beautification problem in the unified sparse coding framework. Face beautification is a process of minutely changing facial features in order to enhance the attractiveness of a rendered face (see, for example, Non-Patent Document 9 and Patent Document 4). Unlike conventional hole-filling processing (for example, see Non-Patent Documents 6 to 8) that attempts to fill in missing pixels using data that can only be used for the currently captured image, two offline correspondence dictionaries are used. It is characterized by using a large-capacity corpus containing both the reconstructed target image of rendering and many beautiful faces, and a huge image database can be easily accessed through social networks and search engines It became available in the era of accessible big data. During image composition, the following code vectors are obtained simultaneously.
(1) One is a sparse code vector for the first dictionary, which contains available DIBR synthesized pixel data.
(2) The other is a sparse code vector for the second dictionary, which matches well with the first vector up to the constrained linear transformation.

  Here, the linear transformation constraint on the predetermined set is a good match between the perception of the captured target image and the beautiful facial features to improve the attractiveness (attractive attractiveness). A balance can be established. The experimental results of this embodiment described below show a naturally rendered human face with recognizable and improved attractiveness.

1-2. Related Art etc. The problem of gaze mismatch is a well-known problem in video conference systems (for example, see Non-Patent Document 1), and there are many solutions in the prior art documents (for example, Non-Patent Documents 12 to 13, 3-5, patent documents 1-3). A solution at an early stage (for example, see Non-Patent Documents 12 and 13) performs rendering processing based on an image on a viewpoint image captured in stereo, and this processing is intensive in digital calculation. Tend to be. By taking advantage of recent advances in depth sensing technology, more recent proposals assuming a capture camera (see, for example, Non-Patent Documents 3, 4, and 5) are for DIBR synthesis of corrected line-of-sight image synthesis. Provides both texture and depth images. Here, the format of the texture image and the depth image requires compression of the depth image on the transmission side, and has been recently studied (for example, see Non-Patent Documents 14 and 15).

  For real-time implementation, these proposed solutions to the “shield removal hole filling problem” are simple and ad hoc, and tend to work well when the camera viewpoint and the target virtual viewpoint are not far apart. However, due to enhanced immersion during video conferencing, large displays are often used and the capture camera can be located far from the center of the screen. The holes thus obtained are large and the inventors' more sophisticated dual sparse coding scheme may be more beneficial, using a dictionary tailored to a particular subject.

  Since the dictionaries required by our approach are learned offline using available corpora for multiple images, the only complexity during the meeting is the calculation of the appropriate sparse code vector. . This is considered to be possible to perform a moderately sized dictionary in real time using the latest optimization tool such as Lasso.

  Face beautification processing has been studied in the literature for individual still images (see, for example, Non-Patent Document 9), and doing so for human face images in a video conferencing situation is time consistent and real-time. Bring new challenges to implementation. In this embodiment, no experiment has been performed, but the sparse coding approach can achieve both objectives by initializing the sparse code vector at the new time using the optimized vector of the previous moment. it is conceivable that.

1-3. System Configuration of Dialogue Device FIG. 1 is a block diagram showing the configuration of the dialogue device provided with the image processing unit 15 according to Embodiment 1 of the present invention. In FIG. 1, a left camera 11, a right camera 12, an image data acquisition unit 13 having a temporary memory 13m, a line-of-sight correction processing unit 14, an image processing unit 15, an image communication unit 16, a display control unit 17, and the like. And a display 18. Here, the image processing unit 15 includes a reconstruction processing unit 21, a dictionary memory 31 connected thereto, a face beautification processing unit 22, and a dictionary memory 32 connected thereto. The dictionary memories 31 and 32 store dictionary matrices Φ and Ψ learned offline as will be described later in detail. Further, as will be described in detail later, the reconstruction processing unit 21 and the face beautification processing unit 22 are performed by a batch image processing unit 21, and image processing is performed substantially in real time.

  The dialogue apparatus uses two cameras 11 and 12 facing the users on the left and right sides of the display 18. The image data acquisition unit 13 acquires the image data from the two cameras 11 and 12, temporarily stores it in the partial memory 13 m, and then outputs it to the line-of-sight correction processing unit 14. The line-of-sight correction processing unit 14 corrects the line-of-sight correction method based on the known DIBR (the line of sight of the user facing the display direction as if it is directed to the camera direction (substantially the image center), so that the other party's dialog ) And the user's line of sight of the interaction device)), the initial texture and depth mapping are assumed to be available from the viewpoint image data captured by the digital camera, and the line of sight is corrected. The combined image data is synthesized and output to the reconstruction processing unit 21 of the image processing unit 15. Due to self-occlusion, pixel loss in the image synthesized by the DIBR line-of-sight correction method that requires filling occurs. Here, filling of pixels that do not belong to the human face is an orthogonal problem that can be solved by using a general hole filling technique (see, for example, Non-Patent Documents 6 to 8).

  Although the two cameras 11 and 12 are provided in the interactive apparatus according to the present embodiment, the present invention is not limited to this, and image processing may be performed after imaging with one camera.

  In this embodiment, the main component is an image processing unit 15 for solving a self-occlusion hole filling problem and a face beautification problem for humans together using a unified dual sparse coding framework. The image processing unit 15 includes a reconstruction processing unit 21 and a face beautification processing unit 22. The image processing unit 15 uses dictionary memories 31 and 32 respectively including two dictionary matrices Φ and Ψ that are learned using two offline learning sets that can be collected a priori.

  The dictionary matrix Φ in the dictionary memory 31 used for reconstruction of face image data is learned using a specific face picture of the current person in the video conference. This is observed that the space stretched by the appearance of the face is relatively small and the space stretched on a specific surface is much smaller. Therefore, by using such a dictionary matrix Φ, a prior probability useful for obtaining a high-quality face reconstruction can be derived.

  In addition, the dictionary matrix Ψ in the dictionary memory 32 used for the beautification of the face is, in the case of the present embodiment, a general beautiful face collection of beautiful face images of Asian movie stars from the Internet. Learned using sets. Through it, the discriminative nature of sparse representation can be used to beautify the face. The atom (image data of the minimum expression) that gives the most compact expression should be given priority as a candidate for beautification and is preferable. That is, face beautification can be expressed as a linear combination of these atoms. Furthermore, the limited linear transformation matrix L is used to constrain the level of beautification, i.e. the deformation should be small. The reconstruction process and face beautification process can be performed iteratively to obtain better results.

  Details of the reconstruction process and the face beautification process of the image processing unit 15 will be described later. The image data processed by the image processing unit 15 is output to the image communication unit 16, converted into predetermined moving image data, and then transmitted to another interactive device. The image communication unit 16 also receives moving image data from the other party's interactive device, and both moving image data are displayed on the display 18 by the display control unit 17.

1-4. Combination of hole filling and face beautification processing Details of the hole filling and face beautification processing executed by the image processing unit 15 will be described below. The inventors have proposed that a successful transformation must meet the following three criteria:
(1) Fidelity: The hole in the face image synthesized by the DIBR line-of-sight correction method needs to be filled as a natural human face.
(2) Attractiveness (attractiveness): It is necessary to enhance the attractiveness of a face having a local characteristic that the face subjected to rendering processing is “beautiful”.
(3) Recognizable: Only a minute deformation needs to be performed on the image data of the original face so that the target subject can be clearly recognized.

The first and second criteria can be formulated as a dual sparse coding problem. Here, the patch-based overcomplete dictionary is learned from the current human example to characterize the current facial structure domain. Also, a feature-based overcomplete dictionary is learned from a set of beautiful faces of movie stars to characterize a plurality of beautiful face feature regions. The third criterion can be met by limiting the type of linear transformation that performs different facial features to limit the degree of beautification. All of these criteria can be put into a unified optimization framework to get the final result.
1-4-1. Dual dictionary learning.
How to derive an appropriate dictionary matrix Φ, Ψ for dual sparse coding is an important problem in the proposed scheme. Two independent dictionary matrices Φ and Ψ are learned for face reconstruction and face beautification, respectively.
1-4-1-1. Dictionary learning for reconstruction.
For face reconstruction, the object of the present embodiment is to fill a hole exposed in the synthesized face image by the DIBR line-of-sight correction method. In the prior art document, it has been shown that image reconstruction using a local patch is very popular and highly effective (for example, see Non-Patent Documents 17 and 18). Therefore, we propose a patch-based local adaptation method for learning dictionary matrices Φ and Ψ for face reconstruction. All patches having overlap are extracted from a plurality of face image data examples given by the current person. Then, the collected patches are classified into a plurality of clusters having the same geometric structure using the K-means clustering method, and each cluster is modeled by learning a compact sub-dictionary. More specifically, for a certain cluster i including n _i image patches to be encoded, a plurality of patch vectors are stored in a matrix indicated by x _i . Then, learning the adaptation sub dictionary matrix [Phi _i most relevant to the matrix x _i by applying principal component analysis method (PCA) to the matrix x _i. The PCA generates an eigenvector dual dictionary matrix Φ _i that is a covariance matrix whose atom of the dictionary matrix Φ is a matrix x _i .

  Since multiple patches in a cluster are similar to each other, the sub-dictionaries do not have to be overcomplete, but because all sub-dictionaries characterize all possible local structures of the face, Combined to build an overcomplete dictionary (see, for example, Non-Patent Document 19).

1-4-1-2. Dictionary learning for face beautification.
Even though multiple faces are objects with great diversity, they are composed of several basic features such as eyes, eyebrows, nose and mouth. The prior art document shows the importance of facial features in the beauty determination task (see, for example, Non-Patent Document 16). Take a feature-based dictionary learning approach for facial beautification. When a front face is given as input, first a set of face landmarks is identified (see, for example, Non-Patent Document 21). Next, six classes of facial features including the left eye, right eye, left eyebrow, right eyebrow, nose, and mouth are extracted according to the position of the landmark. In practical implementation, do not beautify the mouth. This is because it usually moves fast and frequently during a video conference. Then, a sub-dictionary for each feature is constructed.

Specifically, for the current face feature vector x _i , a feature matrix A _i of a sufficient example of the following equation is collected as a learning sample from a beautiful face data set.

  Light normalization is performed on the learning samples to cancel the effects of different lighting conditions. The inventors propose the following model for light normalization.

Here, min () and max () represent a minimum value operator and a maximum value operator, respectively. The learning sample itself after light normalization is used as a basic element of the beautification dictionary matrix Ψ in the dictionary memory 32.
1-4-2. Combined sparse coding.
In this embodiment, the overall objective is achieved by finding the sparse code vectors α, β for the two learned dictionary matrices Φ, Ψ, similar to the linear transformation matrix L, using the following energy function minimization: To do.

  Here, the first two terms of the objective function are standard sparse coding for reconstructing the face vector x related to the dictionary matrix Φ of a specific person. The third and fourth terms perform sparse coding to represent a vector Φα of reconstructed face image data related to a beautiful face dictionary matrix Ψ for the purpose of face beautification. Here, the relationship between the two sparse code vectors α and β is established via a linear transformation matrix L. Note that the linear transformation matrix L indicates transformation from a reconstructed face to a beautiful face synthesized. By constraining the linear transformation matrix L, the degree and type of beautification performed to meet the discriminability criteria can be limited.

The objective function is not a convex function (a function having a maximum or a minimum) with respect to the sparse code vectors α and β and the linear transformation matrix L. However, if other variables are fixed, the convex function is applied to one variable. It becomes. Therefore, an alternate procedure is used repeatedly to optimize these variables. Specifically, to deal with the objective function in equation (2), the objective function is separated into three problems:
(1) Sparse coding for the current face,
(2) Sparse coding for beautiful faces,
(3) Separated into mapping function update. This procedure is repeated until convergence or until a predetermined maximum number of iterations T is reached. In the following, the initialization process, the three sub-problems and their optimal solutions will be described.

1-4-2-1. Initialization process.
By solving the problem of the following equation, a coefficient vector α of the initial sparse representation is obtained.

Here, λ ₁ is a Lagrangian multiplier, and the optimum solution of the coefficient vector α is a fast l1 minimization algorithm known as an augmented Lagrangian method (ALM) (see, for example, Non-Patent Document 20). Can be solved effectively and efficiently. The linear transformation matrix L is then also initialized as a scalar matrix for the purpose of global linear combination of the current facial features and the star facial features. Specifically, in order to limit the level of beautification, the linear transformation matrix L is initialized as L = 0.4I. Here, I is a unit matrix.

1-4-2-2. Optimization for β.
The optimization problem related to the coefficient vector β is expressed by the following equation using the matrix obtained by initializing the diagonal matrix L for performing linear transformation and the derived coefficient vector α.

Here, λ ₂ is a Lagrange multiplier. Finally, a sparse representation coefficient vector β can also be obtained using ALM as previously done for the coefficient vector α.

1-4-2-3. Optimization for L.
The linear transformation matrix L is updated as a minimization problem of the following equation using the derived coefficient vectors α and β.

  In a practical implementation, the linear transformation matrix L is limited to be a simple linear transformation such as rotation, scaling, and shift that constitutes the set Ls. Thereafter, the problem of optimizing the linear transformation matrix L is transformed so that it can be easily searched using the plurality of transformations defined above. In this way, the degree and type of beautification are restricted, and the recognition level is maintained in the face beautification process.

1-4-2-4. Optimization for code vector α.
By fixing the coefficient vector β and the linear transformation matrix L, the sub-problem for the code vector α is expressed by the following equation.

  By performing a simple algebraic operation and deleting constants, the objective function can be reformulated as a standard sparse coding problem as shown in the following equation.

  The above equation can also be efficiently solved by the ALM algorithm. Then, the image processing unit 15 generates and outputs output face image data y using the following equation.

1-6. Conclusion.
Gaze mismatch is a known problem in video conferencing. In the present embodiment, it has been proposed to perform face-beautification filling with a line-of-sight corrected image synthesized by DIBR within one unified sparse coding framework. Assuming the availability of a large-capacity path as a reasonable assumption in the age of big data, the gist of this embodiment is that one is a target human target and the other is two for beautiful face generation. It is to learn offline dictionary matrices Φ and Ψ. As a result, two sparse code vectors can be obtained in real time. One sparse code vector describes pixels synthesized from an image captured by the camera, and the other sparse code vector describes approximate features of a plurality of beautiful faces. The linear transformation that maps from one reconstructed image to another reconstructed image is carefully selected to ensure the recognition of the subject image of the video conference participant. According to the results of experiments by the inventors, it was possible to obtain a naturally rendered face image having improved attractiveness.

Embodiment 2. FIG.
2-1. 1. Introduction This embodiment is an improved extended version of the first embodiment, and is characterized by having the following differences (main improvements). Major improvements are as follows.
(1) Instead of constrained linear transformation, a more general matching criterion between first and second code vectors at feature space distances to ensure the recognition of subject images of participants in a video conference Is introduced.
(2) A new sparse neighbor selection procedure was added to modify the participant's facial contours for added appeal.

In the approach of the present embodiment, the necessary dictionaries are learned offline using the available corpus of images, so that the complexity of the video conference note is only the calculation of the appropriate sparse code vector. This can be done in real time for moderately sized dictionaries using the latest l ₀ -norm optimization tool, as described below.

2-2. Related technology 2-2-1. Face beautification It is disclosed that a beauty score for a human face input by a human evaluator can be predicted using machine learning techniques such as support vector regression (SVR) (for example, Non-Patent Document 16). However, the exact technical definition of the beauty of the human face remains elusive and difficult to understand. Non-Patent Documents 9 and 23 use a similar SVR technique to change to an enhanced attractive human face and determine the appropriate transformation. As an alternative to the SVR technique, a new method called Interactive Evolutionary Computing (IEC) based on a genetic algorithm is used. Instead of facial beautification, for example, Non-Patent Document 27 has devised an algorithm for applying a makeup effect on a female face to enhance attractiveness.

Compared with the related technologies of Non-Patent Documents 9 and 23, the present embodiment is different in the following two main points.
(1) First, in order to construct two dictionary matrices Φ and Ψ offline, a vast database of images on the Internet is used. One dictionary matrix Φ is the face of a TV conference participant. The other dictionary matrix Ψ relates to a beautiful face. In the latter dictionary matrix Ψ, for example, beautiful faces of TV stars and movie stars in China, Japan and Korea from the 20s to 40s are used as learning data. This means that it is not necessary to estimate the beauty score in real time during an actual video conference. Here, only two sparse code vectors need to be identified to map the rendered face image pixels to the beautiful features in the second dictionary matrix Ψ.
(2) Secondly, the purpose of the beautification of the face according to the present embodiment is not to be performed on a single image but to be performed for a moving image encoded and transmitted in a video conference in real time. That is, an overly complex face beautification process that is not suitable for real-time implementation means that it is not suitable for the application. An efficient implementation of the sparse code vector search will be described later in detail.

2-2-2. Sparse representation Sparse representation means the decomposition of a signal into a sparse linear combination of a plurality of atoms in the overcomplete dictionary matrix Φ, Ψ. Here, D is a dictionary matrix as shown in the following equation.

Here, n <K, and each vector d _i is an atom in the dictionary matrix D. The signal vector x (xεR ⁿ ) can be expressed as a linear combination of a plurality of atoms in the dictionary matrix D plus some perturbation ε. That is, it is expressed by the following formula.

here,
And
If it can be achieved at the same time, the expression of the above formula (8) can be said to be “sparse”.

  Dictionary matrices play an important role in sparse representation. In the prior art literature, there is a strong interest in learning a dictionary. For example, machine learning techniques are used to adapt the overcomplete dictionary to the learning data. Examples include the K-SVD method (see, for example, Non-Patent Document 17), supervised dictionary learning (see, for example, Non-Patent Document 29), and an online large-capacity dictionary (see, for example, Non-Patent Document 18). These methods mainly focus on the development of overcomplete dictionaries in a single feature space. Recently, some researchers have suggested learning a dictionary from a combined sparse feature space. For example, Yang et al. (See, for example, Non-Patent Document 31) learn two dictionaries for high resolution (HR) and low resolution (LR) image patch spaces and use the same sparse for each pair of HR and LR patches. We proposed a collaborative dictionary learning method for expression. This method and its followers (see, for example, Non-Patent Document 32) require that two dictionaries used for sparse representation be combined or semi-connected. That is, the atoms in the two dictionaries coming from the observation space and the target space share the same content, but need to be in different expression formats such as LR-HR patches and photo sketch patches. When a new observation image is input, a sparse code related to the observation dictionary is first acquired, and then the sparse code is mapped to the target space based on an equivalent or linear relationship to obtain a final result.

  On the other hand, the method of this embodiment is not subject to these restrictions. In this embodiment, two disjoint dictionaries are constructed to take advantage of the properties of both modeling capabilities and sparse representation discrimination. The first dictionary matrix Φ is used for face reconstruction. This is learned using specific face photos of the current theme of video conferencing. The second dictionary matrix Ψ is used for face beautification, which is typically learned using a set of beautiful faces. Obviously, in this embodiment, both dictionary matrices Φ and Ψ can be seen from the target space. Also, there is no direct relationship between the two sparse codes. The feature space constraint is then used to establish the relationship between the reconstructed patch and the beautification patch. The above feature points are clearly different from learning a combined dictionary according to the prior art.

2-3. System Configuration FIG. 2 is a block diagram showing the configuration of an interactive apparatus provided with an image processing unit 15A according to Embodiment 2 of the present invention. 2, the interactive apparatus according to the second embodiment includes image processing units 15A and 19 instead of the image processing unit 15 as compared with the interactive apparatus of FIG. 1, and the image processing unit 19 includes a face contour adjustment unit. 23, a face contour memory 33 connected thereto, and an image warping processing unit 24. Note that the image processing in the image processing unit 15A is slightly different from the image processing in the image processing unit 15 as described later in detail. Here, as will be described in detail later, the reconstruction processing unit 21 and the face beautification processing unit 22 are performed by the collective image processing unit 21, and image processing is performed substantially in real time.

  The interactive apparatus of the present embodiment can simultaneously capture a texture image (color) and a depth image, such as Microsoft Kinect, at the bottom of the display 18 for the purpose of low cost and easy deployment. A single depth sensing camera 11A is employed. Image data captured by the camera 11A is input to the image processing unit 15A via the image data acquisition unit 13 and the line-of-sight correction processing unit 14 having the temporary memory 13m, which perform the same processing as in the first embodiment.

  Assume that prior to the start of a video conference session, a large set of images, which is a collection of image data of beautiful people with different facial expressions, can be used for offline dictionary learning. The two dictionary matrices Φ and Ψ in the dictionary memories 31 and 32 are learned as follows. That is, first, the dictionary memories 31 and 32 operate for each pixel patch and are used for image complementation, and then operate for each face component (for example, eyes and eyebrows) to beautify the face component. Used for. Here, it is difficult to technically define the concept of human face beauty. Therefore, in this embodiment, it is simply available for men or women via the Internet at the age of 20s to 40s, such as TV stars and movie stars in China, Japan, Korea, etc. Collect face images. In the present embodiment, constituent elements (constituent elements) of face images of TV stars and movie stars are used for learning two dictionary matrices Φ and Ψ. By the reconstruction processing by the reconstruction processing unit 21 and the face beautification processing by the face beautification processing unit 22, the constituent elements of the face are reconfigured so as to beautify the eyes and eyebrows, for example.

  Each pair of texture and depth images captured at the same time is then processed during the video conference as follows. First, foreground / background division processing is performed by threshold processing of a depth image. That is, a pixel having a depth smaller than the preselected value τ is regarded as the foreground. Second, the line-of-sight correction processing based on DIBR is performed on the image data of the pixels identified by the division processing, with the center of the display 18 as the desired virtual viewpoint. Here, the pixels of the texture image from the viewpoint of the captured camera image can be mapped to the corresponding available depth image pixels given the virtual viewpoint. The intrinsic intrinsic and non-essential extraneous matrices required for the DIBR to be mapped can be derived using the standard camera 11A. Due to self-occlusion and missing depth pixels in the captured depth image, there are missing parts in the foreground pixels synthesized in the DIBR-based process that requires filling. For example, instead of using a general hole filling (referred to as inpainting) technique disclosed in prior art documents such as Non-Patent Documents 6, 9, and 10, in this embodiment, hole filling and beautification of face components are performed. Perform processing to combine and reconstruct a visually satisfied face. Next, the face contour adjustment unit 23 optionally adjusts the face contour using the sparse adjacent selection procedure using the face contour memory 33, and the image warping processing unit 24 stores the reconstructed and line-of-sight corrected face data. The warping process is executed so that the background of the original face image is included in the original captured face image by replacing the face that does not match the line of sight by the transposition process and the warping process. Further, by reusing the background of the captured image, it is not necessary to complete a background image by occlusion caused by a virtual image. The face image data output from the image warping processing unit 24 is output to the image communication unit 16, and the subsequent processing is performed in the same manner as in the first embodiment.

  An important technical contribution in the present embodiment is the reconstruction of face image data after eye gaze correction and a face component beautification process via a unified double sparse coding framework. In the present embodiment, two offline learned dictionary matrices Φ and Ψ are given, and two sparse code vectors α and β are searched in cooperation for the two dictionary matrices Φ and Ψ. Here, the dictionary matrices Φ and Ψ are used to describe the pixels of the face image synthesized by the observed DIBR and improve the proximity to the components of the beautiful face.

FIG. 3 is a diagram for explaining a search for a dual code vector executed in the image processing unit 15A of FIG. In FIG. 3, geometrically, cooperative optimization is illustrated. As shown in FIG. 3A, when the observation vector x is given, the face reconstruction vector Φα needs to be close to the l ₂ -norm space in the black circle 41. Further, the face beautification vector Ψβ needs to be close to the vector Φα within the distance (region 42) of the feature space. When the solution is obtained individually, the search area of the sparse code vector β is one set when one sparse code vector α is given, rather than the set of all areas 42 having the center in the circle 41. Region 42. As shown in FIG. 3B, one irregularly shaped region 42 can be approximated by a quadratic function.

In FIG. 3, when an observation vector x that is incomplete and damaged by noise is given, sparse face reconstruction data close to the observation vector x is obtained in an l ₂ -norm space that becomes a circle in a two-dimensional space. . At the same time, in order to maintain the recognition level of the subject image of the video conference at a desired level, a feature space distance constraint S that limits the amount of face beautification to be performed is set. Mathematically, it means that the face beautification vector ψβ needs to be close to the reconstructed vector ψα in the defined feature space in the region 42 for a given α.

  In the present embodiment, the overall objective function can potentially be minimized by cooperating and simultaneously performing occlusion removal hole filling and face component beautification processing, and optimal code vectors α and β can be optimized. Pairs can be identified. Here, the objective function takes into account the filling of both vectors, the amount of face beautification and the fidelity of sparsity. Geometrically, the search space for this pair of these vectors is the union of the entire circle 42 for code vector α and all regions 42 with centers in the circle for code vector β. is there. On the other hand, when hole filling and face beautification are performed separately, after solving for a fixed code vector α, the only region 42 centered on the reconstructed vector Φα is considered as a search space for the code vector β. In this case, there are far more restrictions. In this embodiment, by performing cooperative optimization, facial components having higher beautification scores than individual optimization can be reconstructed.

2-4. Collaborative processing of face reconstruction and face beautification using dual sparse expression The details of the processing of the reconstruction processing unit 21 and the face beautification unit 22 will be described below. A successful algorithm proposes that the following three criteria must be met:

(1) Fidelity: The hole in the face image synthesized by DIBR needs to be satisfied and completed so that a natural human face can be obtained.
(2) Attractiveness (attractiveness): The rendered face needs to enhance the attractiveness of the face having beautiful face components.
(3) Degree of recognition: It is necessary to perform a process on the original face data with a slight correction so that the subject subject can be surely recognized.

  1 is applied to the reconstruction processing unit 21 and the face beautification processing unit 22 in FIG. 1 so as to satisfy the first and second criteria, and the third criterion limits the amount of face beautification in the face beautification processing unit 22. Can be satisfied by introducing appropriate feature space constraints.

2-4-1. Typical image alignment
FIG. 4 is a diagram showing an example of landmarks used in typical image alignment (positioning) executed in the image processing unit 15A of FIG. In the example of FIG. 4, 103 landmarks detected from the face front image of FIG. 4B are illustrated, and FIG. 4A illustrates a two-dimensional plot thereof.

  Each face data including the test image and the learning image is identified by a set of landmarks so that the outlines of all face components can be accurately identified. As shown in FIG. 4, for example, 103 landmarks are identified using the procedure described in Non-Patent Document 21. In order to avoid the effects of scale shifting and rotation, each sample face image in the learning set is aligned to a predetermined default plane.

2-4-2. Sparse Representation in Face Reconstruction Due to self-occlusion, there will be pixel loss in the face image synthesized with DIBR, and an effective face reconstruction procedure to achieve good filling Necessary. The purpose of the face reconstruction is to obtain a higher quality face vector y from the deteriorated observation vector x, and when this is modeled, it is expressed by the following equation.

  Here, H is a deterioration matrix indicating the position of the hole, and v is a model of a synthesis error of processing based on DIBR due to rounding to a two-dimensional pixel grid after projection and inaccuracy of captured depth pixels. Vector.

  From a statistical point of view, reconstructing face data from a degraded format is essentially an inverse problem with poor settings. The performance of the face reconstruction algorithm depends mainly on the degree to which the prior distribution can be used when solving numerical problems. One common technique for incorporating prior knowledge about images is through a so-called sparse model. Here, the face image is approximated by the sparse linear combination of the elements of the appropriately selected dictionary matrix Φ as follows:

  Here, ε is an approximation error. In order to cooperatively represent the pre-data of the degradation matrix H that generates the observation vector x and the dictionary matrix Φ for sparsely representing the input image vector y, the face reconstruction problem is formulated as The

  Here, λ is a Lagrange multiplier.

An important point of the sparse reconstruction is how to find a space in which an atom of a dictionary matrix in which a face image exhibits high sparsity exists. Face reconstruction can be regarded as an image reconstruction problem. For example, prior art documents 17 and 18 show that image reconstruction using a local patch is very common and very effective. In addition, since the face image is non-stationary and statistical having a sparse representation that varies spatially, this embodiment has an adaptive patch unit that learns and represents local statistics, and is a sparse-based replay. The configuration method is shown. All patches having duplication are extracted from the face image examples given by the participants of the video conference. It has been discovered that facial features vary in many parts throughout the image plane, but the facial microstructure can be represented by a small number of structural primitive source data. These primitive source data are qualitatively similar to the format for simplifying the cell acceptance field. Therefore, a plurality of collected patch data is clustered into a plurality of clusters having the same geometric structure using a K-means clustering method (see, for example, Non-Patent Document 19), and a compact sub-dictionary is learned. To model each cluster. Specifically, for a specific cluster i including n _i image patches to be encoded, a plurality of patch vectors are stacked in a matrix indicated by Y _i . Then, by applying principal component analysis method (PCA) with respect to the patch vector Y _i, train the adaptive sub-dictionary matrix [Phi _i most relevant to the patch vector Y _i. PCA produces a sub-dictionary matrix [Phi _i, Atom subdictionaries matrix [Phi _i is the eigenvector of the covariance matrix of the patch vector Y _i.

Because the patches in the cluster are similar to each other, the sub-dictionary matrix Φ _i need not be overcomplete. However, all sub-dictionary matrices Φ _i are joined together to build a large overcomplete dictionary matrix Φ to characterize all possible local structures of the face.

2-4-3. Sparse representation of face beautification processing The structure of a face image can be interpreted as a connection between the contours of facial components and smooth regions (for example, see Non-Patent Document 28). The face beautification process of the face beautification processing unit 22 includes the following three parts.
(1) Facial component beautification (eg, eye enlargement),
(2) facial contour adjustment (such as facial shape adjustment), and (3) skin smoothness.

  Facial reconstruction can achieve the effect of skin smoothness to a certain extent. Because it is based on multiple overlapping patches. The face beautification process performed by the face beautification processing unit 22 and the face contour adjustment performed by the face contour adjustment unit 23 will be described below.

  First, facial component beautification will be described below. Although the face is an object with great diversity, it is composed of several basic components such as eyes, eyebrows, nose and mouth. Conventional research has shown the importance of facial components in the beauty determination task (see, for example, Non-Patent Document 16). When the reconstructed face data is input, four classes of constituent elements of the left eye, right eye, left eyebrow, and right eyebrow face are extracted according to the position of the landmark. Then, one sub-dictionary is constructed for each component individually as follows.

From an aesthetic point of view, the beauty of facial components is reflected by the shape and size of the components, which is described by the location of the landmarks. Therefore, it is assumed that the beautiful face components have a more beautiful distribution of landmarks. Correspondingly, the present embodiment employs a parameter-based dictionary learning approach for beautifying facial components. Specifically, for the current face component x _i , a sufficient amount of example component X _i is collected from a beautiful face data set as a learning sample. Here, the component X _i is represented by the following equation.

From both the test and learning samples, v _i and
Extract the positions of those landmarks represented by The position parameter of the beautiful component to be learned is used as a basic element of the face beautification dictionary matrix Ψ _i . The face beautification dictionary matrix Ψ _i is expressed by the following equation.

The face beautification process is an atom (vector) in the face beautification dictionary matrix Ψ _i .
To approach. Atom with the closest reconstructed component
The reconstruction vector becomes more beautiful as it approaches. Mathematically, the optimum reconstructed landmark distribution β _i ^* is obtained as follows:

Here, β _i ^(j) indicates the j-th element of the landmark distribution β _i . This adjustment term ensures that the closest learning sample is used when evaluating the beautification of the reconstruction. The position of the landmark beautified can be expressed as follows.

  The calculated landmark distribution is then used to warp the incoming component to its beautified version of

Here, W (•, •) is a warping operator, and Φ _i α _i is a component of the reconstructed face corresponding to the component x _i .

  Another challenge with this embodiment is that the resulting beautified face is just subtle and subtle with respect to the original face so as to keep it robust and definitely recognizable from the original face. It is to achieve the purpose of facial beautification by introducing corrections. In this embodiment, the recognizability of each facial component is examined individually, and visibility constraints are described as follows.

Here, S () is a function that maps a pixel area vector to a vector in a selected feature space using, for example, a scale invariant feature transformation method (SIFT) (see, for example, Non-Patent Document 34). Here, the scale invariant feature conversion method (SIFT) performs scaling, rotation, and conversion to a certain degree (thus, even if a small change is made, the recognition degree is not affected). Γ is a constraint parameter that specifies a desired degree of recognition. In other words, Equation (15) indicates that the l ₂ -norm of the feature vector difference between patches before and after face beautification needs to be not too large. If the l ₂ -norm of the feature vector difference is used as a reference for visibility, it is consistent with a document for object search (see, for example, Non-Patent Document 33).

  Due to visibility constraints, the problem of face reconstruction and beautification of face components can be combined into one unified dual sparse coding framework, as shown in the following equation.

Here, λ ₁ and λ ₂ are Lagrange multipliers, and μ is a predetermined repetition constant.

The first two terms of the objective function are common in sparse coding to construct a face vector x _i for the face-specified dictionary matrix Φ _i . In the third term, a relationship between the two sparse code vectors α _i and β _i is established. Then, the image processing unit 15A generates and outputs output face image data y using the following equation.

  Details of how to efficiently calculate equation (16) will be described later.

2-4-5. Face Contour Adjustment Face contour is executed by the face contour adjustment unit 23 in FIG. 2 and highly affects the attractiveness of a human face. In this embodiment, the following four steps are included, and the basic concept is as follows.

(1) For both the test face and the learned and beautified face, the plane graph is configured with landmarks detected as vertices, and a distance vector corresponding to the length of the side in the graph Is extracted.
(2) For the test face, a modified distance vector corresponding to a more beautiful contour is found using the learned beautiful distance vector.
(3) According to the modified distance vector, a new plane graph is generated to form a new edge length as close as possible to the changed distance.
(4) The beautified face contour is obtained by warping the test face to a new location of the multiple landmark results.

  For the first step, do not calculate distances between all pairs of landmarks, eg as in Non-Patent Document 23, but along the face contours indexed as 1-25 in FIG. Calculate the distance between multiple landmark pairs. The main challenge is how to retrieve the modified distance vector. For example, Non-Patent Document 23 proposes correcting the distance vector using a weighted average of the beauty of K nearest neighbors (KNN) of the face in the face space. However, it does not guarantee that all these fixed K neighboring vectors are the first K most relevant ones for the input vector. If the number of similar nearest neighbors is less than K, KNN introduces irrelevant samples that limit the beautification performance. In contrast, the present embodiment proposes an adaptive and flexible sparse neighbor selection method. It is possible to eliminate this problem by finding a minimal closely related learning distance vector to reconstruct the correction.

  Θ represents a distance dictionary, and atoms in the distance dictionary are composed of distance vectors of beautiful faces to be learned. Next, the sparse representation of the test distance vector d is expressed by the following equation.

here,
Indicates a coefficient vector of sparse representation, and n is the number of atoms in the dictionary matrix. Neighbor selection is then achieved according to the following criteria:

  Here, N (d) is a neighbor of d, and δ is a neighborhood selection function defined as follows.

  Here, τ is a threshold value. Once the neighborhood is determined, the weights of all neighbors of N (d) are calculated as follows:

  Finally, the modified distance vector can be derived as:

2-5. Optimization Algorithm In this embodiment, as formulated by Equation (16), the face removal component filling problem for the human face is cooperated with the face beautification problem in the unified sparse coding framework. To solve. However, the objective function formulated in equation (16) is non-convex (no extrema), and there are constraints on the sparse code vector β _i ′, resulting in a constrained non-convex optimization problem. Arise. This makes it difficult to solve directly.

The net effect of the third adjustment term in equation (16) is to limit the number of principle learning samples in face beautification. That is, sparsity can be promoted when the sparse code vector β _i is constructed. This has the same effect as the l ₀ -norm in sparse coding. Therefore, the optimization problem of equation (16) is relaxed as the following equation.

Here, since all face components share the same objective function, the subscript index i is deleted for easy description. HΦ is described as Φ, and u is a vector of weights u _i for modulating elements in the sparse code vector β so as to substantially satisfy the linear constraint in Equation (16). The vector u needs to be updated because equation (15) is solved iteratively as described next.

  The above objective function is not a convex function with respect to the sparse code vectors α and β. Instead of optimizing the objective function directly, an alternative procedure is adopted to solve the problem more effectively. Specifically, the objective function is separated into two subproblems in order to incorporate it into the objective function of equation (22). That is, it is divided into two sub-problems: optimization with respect to the sparse code vector α and optimization with respect to the sparse code vector β. This procedure is repeated until convergence or until a predetermined number of iterations T is reached, whichever comes first. In the following, the initialization process, two sub-problems and their optimal solution will be described.

2-5-1. Initialization processing By solving the problem of the following equation, an initial sparse code vector (sparse expression coefficient) α is obtained under a fixed sparse code vector β.

  The optimal solution of the sparse code vector α can be effectively and efficiently solved by a fast l1-minimization algorithm known as an extended Lagrangian method (ALM) (see, for example, Non-Patent Document 34).

2-5-2. Optimization regarding the sparse code vector β Using the sparse code vector α derived by the equation (23), the optimization problem regarding the sparse code vector β is expressed by the following equation.

  Here, S () is a general nonlinear mapping, and the face beautification penalty term (first term) in Equation (24) describes a feasible space whose shape in the pixel region is irregular. This is illustrated by region 42 in FIG. This makes it difficult to solve the optimization in equation (24).

  Therefore, the face beautification penalty in Equation (24) having a second-order penalty term centered on the vector Φα is approximated.

FIG. 3 is a diagram for explaining a search for a dual code vector executed in the image processing unit 15A of FIG. As shown in FIG. 3B, it can be seen that the irregularly shaped region 42 is approximated by an ellipse (cross-section of the secondary penalty function). Specifically, instead of describing the quadratic penalty function as a full matrix, the quadratic penalty term is converted to a simpler weighted l ₂ − with “coordinate changes” via the linear transformation matrix L. The norm is described as follows.

Here, W is a diagonal matrix having a weight w _i for the matrix element (i, i).

For the above “change in coordinates”, the following Sylvester's law of inertia is used. Sylvester's law of inertia shows that, via a basic appropriate change, the actual quadratic form Q of n variables can be converted to the diagonal form of the weighted sum of

  The optimization problem can then be reformulated as

  Since the sparse code vector α is now fixed and the sparse code vector is unknown, for convenience of calculation, the vector Φα is turned back to the parameter space to obtain the following equivalent optimization problem.

Here, W ⁻¹ () represents an inverse warping operator.

The l ₀ -norm, which is a weighted sparse term of the sparse code vector β, is still difficult to handle. Given an approximated quadratic form of the above optimization, iterative reweighting least squares (IRLS) is used (see, eg, Non-Patent Document 36) to calculate the weighted l ₀ -norm as a sparse acceleration method. in weighted _{l 2} - replace the norm β ^T Uβ. Where U is another diagonal matrix with weights u _i at element (i, i). The optimization can now be written as

Where U ^t is the IRLS weight matrix at iteration t. The minimization of equation (21) at a given iteration t is now an unconstrained quadratic programming problem with a closed form solution, as shown by the following equation (see, for example, Non-Patent Document 37).

  Here, Λ and Ξ are two matrices.

For the weighted l ₂ -norm β ^T Uβ to promote sparsity, equation (28) needs to be solved multiple times, and for each iteration t, adjust the weights in the IRLS weight matrix U ^t . There is a need. In particular,
Is updated to the following equation using the solution β ^t−1 of the previous iteration t−1 for the parameter ε> 0 selected for numerical stability (for example, see Non-Patent Document 36: reference).

The only remaining problem is how to select the linear transformation matrix L and weight w _i so that the approximation in equation (25) is good. For ease of explanation, if the sparse code vector α is fixed,
Is a vector deviating from the vector Φα. The linear transformation matrix L changes the coordinate system so that the secondary penalty function P (v) is only a simple weighted sum of the individual components. That is, it is expressed by the following formula.

Here, () _i represents the i-th element in the vector.

When a fixed vector α ₀ is given, the vector α _{1 is} perturbed by a small amount. The change in the pixel region from the vector Φα ₀ to the vector Φα ₁ is represented by the vector v ₁ , and the change in the feature vector is represented by the following equation.

Is essentially a sample point using evaluating the secondary penalty function P (). Here, when K is such a sample and K <N, the following equation is obtained.

Assuming that these K vectors v _K are linearly independent, iterative projections can be performed to construct a set of K orthogonal bases that spend a K-dimensional space. Given these basis vectors, a K-dimensional quadratic penalty function P (K) can be designed to pass through these sample points. If we assume that the rest of the dimensions (N−K) also have orthogonal basis vectors, each with a weight w _i that is the average of the K computational weights, we get a complete penalty function.

2-5-3. Optimization for Sparse Code Vector α When the sparse code vector β is fixed, a sub-problem for optimizing the sparse code vector α is expressed by the following equation.

  If we remove the constants using simple algebra, the above objective function can be reformulated as follows:

  This has the same form as fact (29). Therefore, it can be seen that it is effective to solve the problem by an iterative reweighting least squares algorithm.

D. Speed Up and Temporal Consistency The inventors contemplate an optimization algorithm for application to video conferencing interaction devices, not just operating on a single image. In this approach, the calculated solution vectors α ^t-1 and β ^t-1 of the previous frame are used in the weighting matrix U ^t during the IRLS optimization process to solve the vectors α ^t and β ^t of the current frame ^t. Can be used for initialization. The quadratic function parameters L and w in equation (18) can also be reused or calculated (refined) for further accuracy. The reuse of this parameter has two distinct advantages:
(1) First, the number of iterations required for IRLS during the optimization process can be greatly reduced, thus speeding up the face reconstruction and beautification process.
(2) Second, as shown in the experiment, temporal consistency between all frames can be achieved by selecting similar beautifying face components.

2-6. Experiment The experiment and experimental results are presented to demonstrate the system performance of the proposed image processor 15A of FIG.

  FIG. 4 is a diagram showing an example of landmarks used in typical image alignment (positioning) executed in the image processing unit 15A of FIG. In the experiment setting of the inventors, as shown in FIG. 4, the inventors captured eight facial images and used them as test samples using the Kect camera 11 </ b> A arranged at the lower part of the display 18. As is clear from FIG. 4, the angle difference between the face image captured by the camera 11 </ b> A and the face image that has been line-of-sight corrected so as to be observed from the center of the display 18 is known from the prior art document (for example, see Non-Patent Document 2). Note that it is much larger than the system).

  First, dictionary learning will be described below. In the experiment, the face reconstruction process is based on a specific face data set consisting of the area of the photographic viewpoint and other frontal photos of the current person with a digital camera of the face. The dictionary matrix Φ for face reconstruction has a sensible patch. In the experiments of the inventors, 5 × 5 small patches for learning the dictionary matrix Φ were overlap-extracted. The face beautification process relies on an offline beautiful face data set. The data set was collected from the web, including 600 samples for males and females, respectively. In order to reduce the effects of age, skin color and other unrelated factors, the inventors have developed a beautiful face learning set, without glasses, from the forties in their 40s in front to age Chinese, Japanese, and Korean TV stars and movie stars were selected and their face image data was collected. The inventors used an algorithm (see, for example, Non-Patent Document 21) for detecting a face region and identifying 103 landmarks as shown in FIG. All star faces were normalized by the distance between the test face pupils and aligned so that the two eyes were in the same position. Male and female beautification faces were used as a unique dictionary matrix Ψ.

  The dictionary matrix Ψ for beautifying the face is based on the face components. For eyes and eyebrows, a plurality of masks were generated with convex polygons formed by adjacent landmarks. Here, the nose landmarks did not form an adjacent polygon. Thus, the nose mask is defined by a triangle covering the upper and lower landmark points.

  Next, captured face image data with a small viewing angle will be described below. In our experiments, for completeness, we also examined the smaller angle case. Here, the self-shielded pixel was simulated by random insertion of impulse noise into the view image at the center of the screen. In actual experiments, 30% of pixels were inserted randomly.

  5A is an image processing result by the image processing unit 15A of FIG. 2 and is a photograph of the image processing result at a small imaging angle with respect to an expressionless frame. The upper part is an image processing result of only the filling process, and the lower part. Is an image processing result of image reconstruction and face beautification processing. 5B is a result of the image processing by the image processing unit 15A of FIG. 2 and is a photograph of the image processing result at a small imaging angle with respect to a frame with a slightly smiling expression, and the upper stage is an image processing result of only the filling process. The lower row shows the image processing results of image reconstruction and face beautification processing.

  In the perceptual evaluation, the face subjected to line-of-sight correction with DIBR was reconstructed, and the results of image processing for beautification of the face are shown in FIGS. The results of FIGS. 5A and 5B include two male test samples and three female test samples. As can be seen from FIGS. 5A and 5B, it can be seen that the method which has been proposed visually is able to achieve satisfactory reconstruction results. It can also be seen that all the holes were filled and fine facial details such as contours were reconstructed. Good face reconstruction may be due to the fact that we can build and derive a useful prior distribution from the dictionary matrix Φ for a particular face. While facial features can vary a lot through facial images, it is found that facial microstructure can be represented by a small number of original images. These original images are qualitatively similar in the form of fields that are acceptable in simple cells. Thus, using many patches extracted from several learning face images, the inventors can learn a dictionary matrix Φ that can well represent test face images.

  From the lower results of FIGS. 5A and 5B, we can see that all test samples have beautified facial components. All samples have eyes that are enlarged to make them look more beautiful. The first four samples have eyebrows more prominently. The second to fourth test samples have a more beautiful face shape with the face contour adjusted. Further, wrinkles around the eyes are removed from the second test sample. A reconstructed face has a dark tint around the eyes due to blocking the light on the nose and hair. The beautification aspect eliminates this effect in order to make the subject appear more active. More importantly, it should be noted that there is still a small amount of gaze mismatch in the face dedicated to reconstruction. Here, the beautification process achieves perfect line-of-sight correction. The difference between the original face beautification is very subtle and thus the similarity between the two faces is unmistakable. However, subtle changes clearly have a noticeable effect on the appeal of these aspects. Video conferencing systems should consider the issue of time consistency.

  Next, statistical evaluation will be described below.

  Because beauty is subjective, in order to select an observer and vote above, the inventors posted a pre-beautified and post-beautified image pair on a web page. Specifically, face image pairs of the same person (before and after beautification in a random order) were announced, and the user was requested to select “left” or “right” as a favorite image. This is basically the two alternative forced selection (2AFC) methods described in, for example, Non-Patent Document 38. This method of comparing images is a method that is less sensitive to noise due to the evaluation of quality, such as the mean opinion score (MOS).

  FIG. 6 is a table showing the voting results for the front image whose face is beautified by the image processing unit 15A of FIG. The inventors have collected votes from 37 observers and the results are summarized in FIG. For example, as disclosed in Non-Patent Document 39, the inventors have assumed a null hypothesis that the selection of a person's original facial image and beautified version is uniformly random, and Pearson's Chi-square test (for example, see Non-Patent Document 40) was performed, and the p value of each image pair was calculated. As a rule of thumb, the null hypothesis is rejected if p <0.05. As shown in FIG. 6, viewers tend to select beautified images (small p-values) of three female subjects (B1, B2, C1, C2, D1, D2) with high likelihood. Thus, the null hypothesis that the viewer has no likelihood between the original image and the beautified image can be rejected. In the case of male subjects (A1, A2), it is difficult to distinguish the likelihood of the viewer. Men's beautification becomes subjective and difficult to achieve consensus.

  FIG. 7 shows the result of image processing by the image processing unit 15A of FIG. 2, which is the result of integrating the face beautified face image into the captured image, the upper part is the original captured face image, and the lower part is It is an integrated face image. FIG. 7 shows an integration result of two test samples. The reconstructed complete face image retains the front and back integration and facial expression as well as line-of-sight matching.

  As described above, the line-of-sight mismatch is a known problem in the video conference system. In the present embodiment, however, the line-of-sight corrected face image synthesized by DIBR using the dual sparse code vector is used. The goal is to provide a virtual reality image of immersive communication that feels like a real experience that is perceptually superior to reality itself, by jointly executing hole filling and face beautification. Assuming the availability of large-capacity corpus data, the key idea is to train the two dictionary matrices Φ, Ψ offline (for the target video conferencing and for the beautifying face). Two sparse code vectors can be determined. One sparse code vector is a sparse code vector for images synthesized with available DIBR, and the other is a feature space so as to establish an acceptable level of cognition while approaching a beautified face. It is a good match with the first vector for distance. Further, the contour of the subject's face is narrowed down as necessary via a sparse neighborhood selection procedure. The inventors have shown through extensive experiments that it is possible to synthesize faces that look natural with enhanced attractiveness.

  As described in detail above, according to the present invention, in the image processing apparatus for an interactive apparatus, correction is performed so that the line of sight of the conversation person coincides, and the problem of “occlusion” is simultaneously solved and face beautification is also performed. be able to.

11 ... Left camera,
11A ... Camera,
12 ... Right camera,
13: Image data acquisition unit,
14: Gaze correction unit,
15, 15A, 19, 20 ... image processing unit,
16 ... image communication part,
17 ... display control unit,
18 ... Display,
21 ... Reconfiguration processing unit,
22 ... Face beautification processing unit,
23. Face contour adjustment unit,
24. Image warping processing unit,
31, 32 ... Dictionary memory,
33: Face outline memory.

Claims (8)

A first dictionary configured by clustering after extracting a plurality of patches from a plurality of face image data, and storing a first dictionary matrix (Φ) for face reconstruction processing for hole filling processing generated by line-of-sight correction Memory,
A second dictionary memory configured by clustering a plurality of components of a beautiful face from a plurality of face image data and storing a second dictionary matrix (Ψ) for face beautification processing;
Based on the input face image vector (x), a first sparse code vector (α) for the first dictionary matrix (Φ) and a second sparse code vector for the second dictionary matrix (Ψ) (Β) and a linear transformation matrix (L) from the reconstructed face image data to the simple image data subjected to the face beautification, for the face reconstruction and the face beautification By obtaining respective optimum values of the first sparse code vector (α), the second sparse code vector (β) and the linear transformation matrix (L) when minimizing the objective function of And image processing means for generating and outputting face image data obtained by performing face reconstruction processing and face beautification processing on the face image vector (x) to be output. The objective function includes a warping operator W (A, B) that warps the input face image data (A) to beautified face image data (B) instead of the linear transformation matrix (L). A mapping function S (•) for mapping a pixel region vector to a vector in a selected feature space using a scale invariant feature transformation method;
The image processing means respectively optimizes the first sparse code vector (α) and the second sparse code vector (β) when the objective function for the face reconstruction and the beautification is minimized. 2. The image according to claim 1, wherein face image data obtained by performing face reconstruction processing and face beautification processing on the input face image vector (x) is generated and output by obtaining a value. Processing equipment. The objective function is expressed as
Here, μ is a predetermined constant, u is a weight vector for modulating the second sparse code vector (β) to satisfy a predetermined linear constraint,
The image processing means obtains the first sparse code vector (α) and the second sparse code vector (β) using the above formula, and performs face reconstruction on the input face image vector (x). The image processing apparatus according to claim 2, wherein face image data subjected to configuration processing and face beautification processing is generated and output.   A face image vector (x) is output by correcting the line of sight of the face image data to be captured so that the line of sight of the face of the face image data matches the line of sight of the conversation person. The image processing apparatus according to claim 1, further comprising a line-of-sight correction processing unit.   A face contour that is provided at a subsequent stage of the image processing means and outputs face image data adjusted by adjusting the contour of the face using a plurality of face contour data for the face image data from the image processing means The image processing apparatus according to claim 1, further comprising an adjusting unit.   An image that is provided after the face contour adjusting unit and warps the face image data from the face contour adjusting unit so as to include the background of the captured face image data and outputs the face image data. 6. The image processing apparatus according to claim 5, further comprising warping processing means. Transmitting / receiving means for receiving face image data from the other party's dialogue device and transmitting the face image data of the dialogue device to the other party's dialogue device;
An interactive device comprising a display unit for displaying each face image data,
The image processing apparatus according to any one of claims 1 to 6, comprising:
The dialogue apparatus characterized in that the transmission / reception means transmits face image data from the image processing device to a dialogue apparatus of the other party. A first dictionary configured by clustering after extracting a plurality of patches from a plurality of face image data, and storing a first dictionary matrix (Φ) for face reconstruction processing for hole filling processing generated by line-of-sight correction Memory,
An image processing apparatus comprising: a second dictionary memory configured by clustering a plurality of components of a beautiful face from a plurality of face image data and storing a second dictionary matrix (Ψ) for face beautification processing An image processing method for
Based on the input face image vector (x), the image processing apparatus performs the first sparse code vector (α) for the first dictionary matrix (Φ) and the second dictionary matrix (Ψ). An objective function comprising a second sparse code vector (β) and a linear transformation (L) from the reconstructed face image data to the simplified face data subjected to face beautification, wherein the face reconstruction and Obtaining respective optimum values of the first sparse code vector (α), the second sparse code vector (β), and the linear transformation (L) when minimizing the objective function for the beautification of the face And generating and outputting face image data obtained by performing face reconstruction processing and face beautification processing on the input face image vector (x).