JP2009246408A - Interaction device, image processing module, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an interaction device capable of easily achieving a natural interaction environment, where the lines of sight of speakers coincide, in interactive communication, and to provide an image processing module, an image processing method and a program. <P>SOLUTION: The interaction device has a camera 11 as an imaging means, an image processing module 13, a display section 32, an image communication section 33 and the like. The image processing module 13 changes the positions of an iris and a pupil by changing the pixel of the palpebral fissure region (an exposure region of an eyeball) of a person included in image data while the line of sight of the person included in inputted image data points to the front of the image data. The image processing module 13 has a face recognition section 14, a line-of-sight determination section 15, and a line-of-sight change section 16. The face recognition section 14 extracts information on a vector (face vector) prescribing the direction of the whole of a face, based on the result of face recognition processing. The line-of-sight change section 16 changes the pixel in the palpebral fissure region so that the line of sight of the person points to the front of the image data, when it is determined that the face points to a display screen based on the direction of the face vector. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an interactive device, an image processing module, an image processing method, and a program that are used for a videophone or a video conference.

  One of the problems in interactive communication systems such as videophones and videoconferencing is the problem of disagreement between the lines of dialogue between the interlocutors.

  In general, when using an interactive communication system, a user tries to match his / her line of sight with an image of a partner displayed on a display device. However, when the normal direction of the display screen of the display device is different from the optical axis direction of the lens of the camera that captures the user, when the user aligns the line of sight with the image of the other party, the user image captured by the camera is It is not a front image. In this case, although the user aligns the line of sight with the image of the other party, the image that the user is turning the line of sight is displayed on the display device on the other side.

  Conventionally, there is a technique disclosed in Japanese Patent Application Laid-Open No. 2004-326179 (Patent Document 1) as a technique for matching the lines of sight of interlocutors in this type of interactive communication system.

The image processing apparatus disclosed in Patent Document 1 includes approximate image generation means for generating an approximate image when a user's head image input from a camera is viewed from a predetermined size and direction, and includes a plurality of approximate image generation means. When performing a videoconference with simultaneous display of multiple interactors, it is possible to unify the size of each conversational person's head, and to create an image in which each conversational person's face appears facing the camera. It can be generated automatically.
JP 2004-326179 A

  However, in the conventional technology, an image in which each conversation person's face is facing the camera is generated regardless of the direction in which each conversation person faces. For this reason, a very unnatural image is generated for an interlocutor whose posture is clearly not intended to face the camera. Further, in the conventional technique, since it is necessary to change the direction of the whole face of each interlocutor, complicated image processing must be performed, and the image processing takes time.

  The present invention has been made in consideration of the above-described circumstances, and an interactive apparatus, an image processing module, and an image processing method that can easily realize a natural interactive environment in which the lines of sight of the interrogators match in interactive communication. And to provide a program.

  An image processing module according to the present invention is an image processing module used in an interactive communication system in order to solve the above-described problem, and includes at least the orientation of the face from image data including an inputted person's face. Face recognition means for extracting information on a face vector defining the position, information on the position of an eyelid in the face, and information on an iris pixel of the face, and the face based on the information on the face vector. A line-of-sight determining unit that determines whether or not the display unit is facing the display screen; and if it is determined that the face is facing the display screen of the display unit, the line of sight of the person faces the front of the image data , Line-of-sight changing means for changing pixels in the eyelid fracture region.

  Further, in order to solve the above-described problem, the interactive apparatus according to the present invention includes at least information on a face vector that defines the orientation of the face from display means and image data including the face of the input person, Based on face recognition means for extracting information on the position of the eyelids in the face and information on iris pixel of the face, and on the basis of the information on the face vector, the face faces the display screen of the display means. A line-of-sight determination unit that determines whether the face is facing the display screen of the display unit, and pixels in the eyelid region so that the line of sight of the person faces the front of the image data Line-of-sight changing means for changing the image, and the display means displays the image data in which the pixels in the eyelid region received from the line-of-sight changing means are changed.

  On the other hand, in order to solve the above-described problem, the image processing method according to the present invention includes at least information on a face vector that defines the orientation of the face from image data including the face of the input person, Extracting information about the position of the eyelid and pixel information of the iris of the face, and determining whether the face is facing the display screen of the display means based on the information of the face vector And, when it is determined that the face is facing the display screen of the display means, changing the pixels in the eyelid region so that the line of sight of the person faces the front of the image data It is the method characterized by this.

  Further, in order to solve the above-described problem, the program according to the present invention causes a computer to acquire at least information on a face vector that defines the orientation of the face from the input image data including the face of the person, and the face. Based on the face recognition means for extracting the information on the position of the eyelids in the eye and the information on the iris pixel of the face, it is determined whether or not the face is directed to the display screen of the display means And a line of sight that changes pixels in the eyelid region so that the line of sight of the person faces the front of the image data when it is determined that the face is facing the display screen of the display unit This is a program for functioning as a changing means.

  According to the interactive apparatus, the image processing module, the image processing method, and the image processing program according to the present invention, it is possible to easily realize a natural interactive environment in which the lines of sight of the interlocutors coincide in interactive communication.

  Embodiments of an interactive apparatus, an image processing module, an image processing method, and a program according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a schematic overall configuration diagram showing an embodiment of an interactive apparatus 10 according to the present invention. The interactive device 10 can be applied to, for example, a personal computer or a mobile phone.

  As shown in FIG. 1, the dialogue apparatus 10 includes a camera 11 as an imaging unit, an image data acquisition unit 12, an image processing module 13, a display control unit 31, a display unit 32, and an image communication unit 33. Have.

  The user can use the interactive device 10 to use a videophone or a video conference with a conversation partner using another interactive device 10 connected to the interactive device 10 through a network so that image communication is possible.

  The camera 11 captures an image including a user's face using the interactive device 10 and generates image data. As this camera 11, for example, a camera using a CMOS image sensor or a CCD image sensor can be used.

  The image data acquisition unit 12 includes image data of an image including the face of the conversation partner acquired from the image communication unit 33 (hereinafter referred to as reception image data), and image data of an image including the user's face acquired from the camera 11 ( (Hereinafter referred to as transmission image data) is input to the image processing module 13.

  The image processing module 13 changes the pixels of the human eyelid area (exposed area of the eyeball) included in the image data so that the line of sight of the person included in the input image data faces the front of the image data. Change the position of the iris and pupil.

  Note that the fact that the person's line of sight is “facing the front of the image data” is synonymous with the fact that the person's line of sight is “facing the camera 11”.

  The display control unit 31 generates an image from the image data output from the image processing module 13 and causes the display unit 32 to display the image. The image data given to the display control unit 31 includes at least received image data (image data including the face of the conversation partner) output from the image processing module 13.

  In the following description, an example in which the image data given to the display control unit 31 is both received image data and transmitted image data (image data including a user's face) will be described. In this case, whether the user is performing image processing by looking at the transmission image displayed on the display unit 32 so that his / her line of sight when viewing the display screen of the display unit 32 faces the camera 11. Can be confirmed.

  The display unit 32 is configured by a general display output device such as a liquid crystal display or a CRT display, for example, and displays various images according to the control of the display control unit 31.

  The image communication unit 33 implements various information communication protocols according to the network form. The image communication unit 33 connects the interactive device 10 and another interactive device 10 according to these various protocols. For this connection, an electrical connection via an electronic network can be applied. The image communication unit 33 transmits / receives image data to / from another interactive apparatus 10 via this connection. Here, the electronic network means an entire information communication network using telecommunications technology. In addition to a LAN (Local Area Network) and the Internet network, a telephone communication line network, an optical fiber communication network, a cable communication network, and a satellite communication network. Etc.

  Next, the image processing module 13 will be described in detail.

  The image processing module 13 includes a face recognition unit 14, a line-of-sight determination unit 15, and a line-of-sight change unit 16.

  The face recognition unit 14 performs face recognition processing on the input image data, and extracts information on the position of the eyelid and pixel information on the eyelid region. Conventionally, various face recognition processing methods are known, and any of these methods can be used. Further, the face recognition unit 14 extracts information on a vector that defines the orientation of the entire face (hereinafter referred to as a face vector) based on the result of the face recognition process.

  The line-of-sight determination unit 15 determines whether the face of the person included in the image data is facing the display screen of the display unit.

  The line-of-sight change unit 16 includes a black eye information acquisition unit 17, a model generation unit 18, a model position determination unit 19, and an eyelid region change unit 20.

  The black eye information acquisition unit 17 acquires pixel information of the iris and pupil (including information on the size and color of the iris and pupil) from the face recognition unit 14.

  The model generation unit 18 generates an iris model and a pupil model based on information on iris and pupil pixels.

  The model position determining unit 19 determines the arrangement positions of the iris model and the pupil model so that the person's line of sight faces the front of the image data based on the face vector information.

  The eyelid region changing unit 20 generates image data in which the user's line of sight faces the front of the image data by changing the pixels in the eyelid region, and this image data is displayed on the display control unit 31 and the image communication unit 33. To give.

  Next, an example of the operation of the interactive apparatus 10 according to this embodiment (including the operation of the image processing module 13) will be described.

  FIG. 2 shows image data including the user's face generated by the camera 11 in order to easily realize a natural dialogue environment in which the lines of sight of the dialoguers match in the interactive communication by the dialogue apparatus 10 shown in FIG. It is a flowchart which shows the procedure at the time of changing the eyes | visual_axis of as needed. In FIG. 2, a symbol with a number added to S indicates each step of the flowchart.

  In the following description, an example in which the installation position of the camera 11 is at the upper left when viewed from the user will be described.

  First, in step S <b> 1, the image data acquisition unit 12 receives image data including the user's face from the camera 11 and inputs this image data to the image processing module 13.

  Next, in step S <b> 2, the face recognition unit 14 performs face recognition processing on the input image data, and extracts information on the position of the eyelid in the user's face and information on the pixels in the eyelid region. The pixel information in the eyelid region includes information on the iris, pupil, and sclera (so-called white eye) pixels. Information on each pixel includes information on the position, size, and color of each object.

  Further, the face recognition unit 14 extracts face vector information that defines the orientation of the entire face based on the result of the face recognition process.

  FIG. 3 is a diagram for explaining the face vector extracted from the image data when the user's face direction and line of sight are directly facing the display screen of the display unit 32. The left side of FIG. 3 is image data generated by the camera 11. When the user's face is facing the camera 11, the user's face is facing the front of the image data (the direction from the back to the front of the paper).

  When the user's face is directly facing the display screen of the display unit 32, the face vector is parallel to the normal vector of the display screen. On the other hand, in this case, as shown in FIG. 3, the face vector depends on the installation position of the camera 11 with respect to the normal vector of the image data that defines the front of the image data (the vector that penetrates from the back of the page to the front). With a slightly offset orientation.

  FIG. 4 is a diagram for explaining a face vector extracted from image data when the face direction of the user deviates greatly from a state in which the display screen of the display unit 32 faces straight.

  As shown in FIG. 4, when the user is not looking at the conversation partner (display screen of the display means), the face vector has a direction greatly deviated from the normal vector of the display screen.

  If the user's face is clearly not facing the conversation partner, an unnatural image will be generated if the line of sight is changed. Therefore, it is determined whether or not the user's face is facing the display screen of the display means based on whether or not the face vector is in a predetermined range, and the line of sight is changed only when the face vector is in the predetermined range. It is advisable to perform processing.

  In step S <b> 3, the line-of-sight determination unit 15 determines whether the face of the user included in the image data is facing the display screen of the display unit by determining whether the orientation of the face vector is within a predetermined range. . If the orientation of the face vector is within a predetermined range, that is, if it is determined that the user's face is facing the display screen of the display means, the process proceeds to step S4 to perform a line-of-sight change process. On the other hand, when the orientation of the face vector is outside the predetermined range, that is, when it is determined that the user's face is not facing the display screen of the display unit, the line-of-sight determination unit 15 obtains the image data before the line-of-sight change process. This is given to the display control unit 31 and the image communication unit 33. Proceed to step S8.

  Next, in step S <b> 4, the black eye information acquisition unit 17 acquires iris and pupil pixel information (including iris and pupil size and color information) from the face recognition unit 14.

  FIG. 5 is an explanatory diagram showing an example of how an iris model and a pupil model are generated.

  In step S5, the model generation unit 18 generates an iris model and a pupil model based on information on iris and pupil pixels. As a result, an iris model and a pupil model that match the user's iris and pupil color can be generated, and an iris model and a pupil model that match the size of the iris and pupil on the image data can be generated. .

  The iris model and the pupil model are used when performing image processing so that the user's line of sight faces the front of the image data. The shape of the iris and pupil of the person facing the front is visually recognized as a circular shape by others. For this reason, the model generation part 18 should just produce | generate an iris model and a pupil model so that it may become a substantially circular shape, and does not need to perform complicated shape calculation.

  Next, in step S6, the model position determination unit 19 determines the arrangement positions of the iris model and the pupil model based on the face vector information. More specifically, the model position determination unit 19 arranges the iris model and the pupil model so that the user's line of sight faces the front of the image data based on the difference between the orientation of the face vector and the front of the image data. Determine the position.

  FIG. 6 is an explanatory diagram illustrating an example of a state in which the pixels in the eyelid region of the image data are changed. FIG. 7 is a diagram for describing an example of image data before and after the line-of-sight change process.

  In step S7, the eyelid region changing unit 20 changes the pixels in the eyelid region so that the iris model and the pupil model are arranged at the position determined by the model position determining unit 19, so that the user's line of sight is image data. Is generated (refer to FIG. 7). Then, the eyelid fracture region changing unit 20 gives the image data after the line-of-sight changing process to the display control unit 31 and the image communication unit 33.

  For example, as shown in FIG. 6, first, the pixels in the eyelid fracture region are filled with the sclera color acquired from the face recognition unit 14, then the substantially circular iris model and pupil model are placed in the foreground, Finally, image data with the user's line of sight facing the front of the image data can be generated by the procedure of trimming the iris model and the pupil model according to the eyelid fracture.

  Next, in step S <b> 8, the display control unit 31 takes an image from the image data (image data before the line-of-sight change process received from the line-of-sight determination unit 15 or image data after the line-of-sight change process received from the eyelid region change unit 20). Is generated and displayed on the display unit 32. In addition, the image communication unit 33 sends the image data (image data before the line-of-sight change process received from the line-of-sight determination unit 15 or image data after the line-of-sight change process received from the eyelid region change unit 20) to the other interactive device 10. Send to.

  With the above procedure, the line of sight of the image data including the user's face generated by the camera 11 can be changed as necessary, and a natural conversation environment in which the lines of sight of the interlocutors match in the interactive communication can be easily performed. Can be realized.

  Further, when the dialogue apparatus of the dialogue partner does not include the image processing module 13 of the dialogue apparatus 10 according to the present embodiment, and the received image data received by the image communication unit 33 is image data before the line-of-sight change process. By performing the procedure shown in FIG. 2 on the received image data, the line-of-sight changing process can be performed on the received image data.

  In this case, for example, in step S <b> 1, the image data acquisition unit 12 may receive image data including the face of the conversation partner from the image communication unit 33 and input this image data to the image processing module 13. In step S <b> 7, the eyelid fracture region changing unit 20 only needs to provide the received image data after the line-of-sight changing process to the display control unit 31, and does not need to provide the image communication unit 33.

  Of course, when the dialogue apparatus of the dialogue partner includes the image processing module 13 of the dialogue apparatus 10 according to the present embodiment, the image acquisition unit may directly provide the received image data to the display control unit 31.

  The dialogue apparatus 10 according to the present embodiment can acquire face vector information. For this reason, it is possible to determine whether the person included in the image data is looking at the conversation partner (looking at the display screen) based on the orientation of the face vector. Therefore, according to the dialogue apparatus 10 according to the present embodiment, it is possible to perform the line-of-sight change process only when a person included in the image data is looking at the conversation partner, and the lines of sight between the dialogue persons in interactive communication can be obtained. A consistent natural dialogue environment can be easily realized.

  In addition, the interactive apparatus 10 according to the present embodiment uses only one camera 11 without using a special physical device or synthesizing a plurality of images, and image processing is performed only for pixels in the eyelid region. You can change it. For this reason, the line-of-sight changing process can be performed very simply and at high speed.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment.

  For example, a part or all of the configuration of the image processing module 13 described above can be made to function by executing a program by a computer. In this case, the image processing program stored in the program memory is installed by installing the image processing program from a CD-ROM, a magnetic disk or the like into a program memory of a computer or downloading it to the program memory via a communication line. It can be executed by a computer.

  Further, in the embodiment of the present invention, each step of the flowchart shows an example of processing that is performed in time series in the order described. The process to be executed is also included.

1 is a schematic overall configuration diagram showing an embodiment of an interactive apparatus according to the present invention. In order to easily realize a natural conversation environment in which the lines of sight of the interlocutors coincide in the interactive communication, the line of sight of the image data including the user's face generated by the camera is used as necessary. The flowchart which shows the procedure at the time of changing. The figure for demonstrating the face vector extracted from image data, when the direction of a user's face and a gaze are looking straight at the display screen of a display part. The figure for demonstrating the face vector extracted from image data when the direction of a user's face deviates significantly from the state which faced the display screen of the display part. Explanatory drawing which shows an example of a mode that an iris model and a pupil model are produced | generated. Explanatory drawing which shows an example of a mode that the pixel in the eyelid region of image data is changed. The figure for demonstrating an example of the image data before and behind a gaze change process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Dialogue device 11 Camera 12 Image data acquisition part 13 Image processing module 14 Face recognition part 15 Gaze determination part 16 Gaze change part 17 Black eye information acquisition part 18 Model generation part 19 Model position determination part 20 Eyelid region change part 31 Display control part 32 Display unit 33 Image communication unit

Claims (11)

An image processing module used in an interactive communication system,
Extract at least face vector information that defines the orientation of the face, information on the position of the eyelids in the face, and information on iris pixel of the face from the image data including the face of the input person. Facial recognition means to
Line-of-sight determining means for determining whether the face is facing the display screen of the display means based on the information of the face vector;
When it is determined that the face is facing the display screen of the display unit, the line-of-sight changing unit that changes the pixels in the eyelid region so that the line of sight of the person faces the front of the image data;
An image processing module comprising: The iris pixel information is
At least information on the position of the iris on the face, information on the size of the iris, and information on the color of the iris,
The line-of-sight changing means includes
A substantially circular iris model is generated based on the information of the iris pixels, and the arrangement position of the iris model is determined based on the face vector information so that the line of sight of the person faces the front of the image data. Configured to be able to change the pixels in the eyelid region so that the iris model is arranged at the arrangement position,
The image processing module according to claim 1. Display means;
Extract from the input image data including the face of the person at least information on the face vector that defines the orientation of the face, information on the position of the eyelid in the face, and information on the iris pixel of the face Facial recognition means to
Line-of-sight determination means for determining whether the face is facing the display screen of the display means based on the information of the face vector;
When it is determined that the face is facing the display screen of the display unit, the line-of-sight changing unit that changes the pixels in the eyelid region so that the line of sight of the person faces the front of the image data;
With
The display means includes
An interactive apparatus that displays the image data in which pixels in the eyelid region received from the line-of-sight changing means are changed. The iris pixel information is
At least information on the position of the iris on the face, information on the size of the iris, and information on the color of the iris,
The line-of-sight changing means includes
A substantially circular iris model is generated based on the information of the iris pixels, and the arrangement position of the iris model is determined based on the face vector information so that the line of sight of the person faces the front of the image data. Configured to be able to change the pixels in the eyelid region so that the iris model is arranged at the arrangement position,
The interactive apparatus according to claim 3. Image communication means configured to be able to transmit and receive image data to and from other interactive devices;
Further comprising
The face recognition means
The image data received from the other interactive device via the image communication means is input.
The interactive apparatus according to claim 3 or 4. The display means includes
Further displaying image data received from the other interactive device in which a pixel in the eyelid fracture region received from the line-of-sight changing means has been changed;
The interactive apparatus according to claim 5. Imaging means for capturing an image including the person's face and generating image data;
Further equipped with,
The dialogue apparatus according to any one of claims 3 to 6. Extract at least face vector information that defines the orientation of the face, information on the position of the eyelids in the face, and information on iris pixel of the face from the image data including the face of the input person. And steps to
Determining whether the face is facing the display screen of the display means based on the information of the face vector;
When it is determined that the face is facing the display screen of the display means, changing the pixels in the eyelid region so that the line of sight of the person faces the front of the image data;
An image processing method comprising: The iris pixel information is
At least information on the position of the iris on the face, information on the size of the iris, and information on the color of the iris,
Changing the pixel in the eyelid fracture region comprises:
Generating a substantially circular iris model based on the iris pixel information;
Determining an arrangement position of the iris model so that the line of sight of the person faces the front of the image data based on the information of the face vector;
Changing the pixels in the eyelid region so that the iris model is placed at the placement position;
Having
The image processing method according to claim 8. Computer
Extract from the input image data including the face of the person at least information on the face vector that defines the orientation of the face, information on the position of the eyelid in the face, and information on the iris pixel of the face Facial recognition means,
Based on the information on the face vector, the line-of-sight determining means for determining whether the face is facing the display screen of the display means, and if it is determined that the face is facing the display screen of the display means, Line-of-sight changing means for changing pixels in the eyelid area so that the line of sight of the person faces the front of the image data;
Program to function as. The iris pixel information is
At least information on the position of the iris on the face, information on the size of the iris, and information on the color of the iris,
The line-of-sight changing means includes
A substantially circular iris model is generated based on the information of the iris pixels, and the arrangement position of the iris model is determined based on the face vector information so that the line of sight of the person faces the front of the image data. Configured to be able to change the pixels in the eyelid region so that the iris model is arranged at the arrangement position,
The program according to claim 10.

Images