JP2018055619A - Image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display device capable of accurately displaying operation of a display object and also displaying an image model suitable for the display object.SOLUTION: An image display device includes: a display part displaying an image; an imaging part imaging a subject and acquiring a subject image; an operation detection part detecting operation of the subject by using the subject image and acquiring attitude information of the subject; a face recognition part performing face recognition of the subject by using the subject image and acquiring attribute information of the subject; and a control part creating an image model of the subject and displaying the image model at the display part. The control part determines based on the attribute information whether or not the subject is a display object at the display part, and upon determination that the subject is the display object at the display part, in accordance with the operation of the subject based on the attitude information, causes the display part to display the image model created based on the attribute information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image display device.

  There is known a virtual try-on device that generates an image of a try-on who virtually tried on clothes and displays the image on a display means even if the try-on does not actually try on clothes.

  Patent Document 1 discloses a moving image photographing means, a posture estimating means for estimating a fitting person's posture reflected in a moving image from the fitting person's three-dimensional body shape data and various posture data, and a smooth and natural image from the time-series estimated posture. Disclosed is a virtual try-on device comprising posture correcting means for generating correct posture time-series data, clothes movement generating means for calculating movement of clothes, and display means for displaying a state in which a wearer has tried on clothes. ing.

JP 2006-249618 A

  The virtual try-on device of Patent Document 1 includes three-dimensional human body shape data that has been measured and stored in advance, and a posture dictionary that has been previously measured and stored various posture data. Is used to estimate the current motion and posture of the try-on. In other words, the virtual try-on device of Patent Document 1 requires a huge amount of measurement in advance in order to estimate the movement of the try-on, and the amount of calculation is also a huge amount. Further, in the virtual try-on device of Patent Document 1, if the posture data stored in the posture dictionary is reduced in order to reduce the amount of calculation, the accuracy of estimation decreases. For this reason, it is difficult for the virtual try-on device of Patent Document 1 to accurately represent the operation of the try-on with a simple configuration, and it is difficult to easily introduce it.

  In addition, the virtual try-on device of Patent Document 1 displays an image of clothes to be tried on an image of a try-on person in the moving image obtained from the moving image photographing means. That is, in the virtual try-on device of Patent Document 1, the image displayed on the display means includes an image of clothes currently worn by the try-on, and an image of clothes to be tried on and an image of clothes currently being worn. May interfere. For this reason, it is difficult for the virtual try-on device of Patent Document 1 to display a try-on image that is truly suitable for a try-on.

  In addition, the virtual try-on device of Patent Document 1 is based on the premise that there is only one try-on, and is assumed to be used as a substitute for a conventional fitting room. That is, the virtual try-on device of Patent Document 1 is not configured to execute processing when a plurality of try-onrs are simultaneously imaged by the moving image photographing means. For this reason, it is difficult to use the virtual try-on device of Patent Document 1 in a situation where a plurality of try-onrs are assumed to be simultaneously imaged by the moving image photographing means.

  This invention is made | formed in view of the above-mentioned situation, and makes it an example of a subject to solve the above problems. That is, an example of the subject of this invention is providing the image display apparatus which can display the operation | movement of a display object accurately and can display the image model suitable for a display object.

  An image display device according to an aspect of the present invention includes a display unit that displays an image, an imaging unit that captures a subject and acquires a subject image, detects an operation of the subject using the subject image, and A motion detection unit that acquires subject posture information, a face recognition unit that performs face recognition of the subject using the subject image, acquires attribute information of the subject, creates an image model of the subject, and displays the display A control unit that displays on the display unit, wherein the control unit determines whether the subject is a display target of the display unit based on the attribute information, and the subject is the display target of the display unit. When the determination is made, the image model created based on the attribute information is displayed on the display unit in accordance with the movement of the subject based on the posture information.

  Preferably, the control unit uses the attribute information to create a three-dimensional CG model as the image model, and the three-dimensional according to the change of the posture information according to the motion of the subject. A display control unit that moves the CG model and displays the CG model on the display unit.

  Preferably, the motion detection unit detects the position of the posture feature point of the subject, the face recognition unit detects the position of the face feature point of the subject, and the control unit detects the position of the posture feature point of the subject. An association unit that associates the posture information with the attribute information based on the position and the position of the face feature point, and the model creation unit includes the posture information and the attribute information associated with the association unit; Is used to create the three-dimensional CG model.

  Preferably, the association unit converts a coordinate system of the position of the posture feature point and a coordinate system of the position of the face feature point into a common coordinate system, respectively, and the position of the posture feature point in the common coordinate system Is matched with the position of the face feature point to associate the posture information with the attribute information.

  Preferably, the control unit further includes a determination unit that determines whether the subject is the display target using the attribute information, and the imaging unit includes the subject image in which a plurality of subjects are reflected. The motion detection unit acquires a plurality of the posture information corresponding to the plurality of subjects, the face recognition unit acquires a plurality of the attribute information corresponding to the plurality of subjects, and the association A unit associates the plurality of posture information and the plurality of attribute information, respectively, and the determination unit includes the posture that meets a predetermined condition among the plurality of posture information and the attribute information associated by the association unit. The information and the attribute information are selected, the subject corresponding to the selected posture information and the attribute information is determined to be the display target, and the model creation unit is selected by the determination unit. The three-dimensional CG model of the subject determined to be the display target by the determination unit is created using the posture information and the attribute information, and the display control unit is configured to display the display by the determination unit. The three-dimensional CG model created by the model creation unit is moved and displayed on the display unit in accordance with the change in the posture information according to the movement of the subject determined to be a subject.

  The image display apparatus according to one aspect of the present invention can display the operation of the display object with high accuracy even with a simple configuration, and can display an image model that is truly suitable for the display object.

Several embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings.
It is a figure which shows the external appearance structure of the image display apparatus which concerns on embodiment of this invention. It is a block diagram which shows the functional structure of the image display apparatus which concerns on embodiment of this invention. It is a figure which shows the to-be-photographed image and attitude | position feature point which are acquired with the image display apparatus which concerns on embodiment of this invention. It is a figure which shows the face image and face feature point which are contained in the to-be-photographed image acquired with the image display apparatus which concerns on embodiment of this invention. It is a flowchart which shows the detailed flow of the image display process performed in the image display apparatus which concerns on embodiment of this invention. It is a figure which shows the display image displayed with the image display apparatus which concerns on embodiment of this invention. It is a flowchart which shows the detailed flow of the correlation process performed in the image display apparatus which concerns on embodiment of this invention. It is a flowchart which shows the detailed flow of the display target judgment process performed in the image display apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Embodiment described below shows some examples of the present invention, and does not limit the contents of the present invention. In addition, all the configurations and operations described in the embodiments are not necessarily essential as the configurations and operations of the present invention. In addition, the same referential mark is attached | subjected to the same component and the overlapping description is abbreviate | omitted.

[1. Configuration of image display apparatus]
FIG. 1 is a diagram showing an external configuration of an image display device 1 according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the image display apparatus 1 according to the embodiment of the present invention. FIG. 3 is a diagram showing subject images and posture feature points acquired by the image display device 1 according to the embodiment of the present invention. FIG. 3A shows a subject image. FIG. 3B shows posture feature points. FIG. 4 is a diagram showing face images and face feature points included in the subject image acquired by the image display device 1 according to the embodiment of the present invention. FIG. 4A shows a face image. FIG. 4B shows facial feature points.

  The image display device 1 is a device that displays images and information. The image display device 1 includes a digital signage device that advertises advertisements to passers-by by displaying images and information, and a virtual fitting device that displays images in which passers-by virtually try on clothing and other clothing items It may be a device that also serves as The image display device 1 may be installed in a display space in a show window where a mannequin or the like is arranged, for example.

  The image includes both a still image and a moving image. In addition, it is assumed that the image displayed by the image display apparatus 1 includes information such as text related to the image as necessary.

  As shown in FIGS. 1 and 2, the image display device 1 includes a display unit 10, a motion sensor 20, and an information device 30. The display unit 10, the motion sensor 20, and the information device 30 are connected to each other so that various types of information can be communicated with each other by wire or wirelessly.

  The display unit 10 is a display that displays an image based on control from the information device 30. The display unit 10 may be configured by various displays such as a liquid crystal display or an organic EL display, for example.

  The motion sensor 20 is a sensor that captures an image of a subject such as a passerby and can detect the operation of the subject in real time. The motion sensor 20 may be configured by a commercially available motion sensor used in a motion capture system. The subject includes animals such as pets accompanied by passers-by as well as persons such as passers-by.

  The motion sensor 20 may be installed in the upper center of the display unit 10 in the image display device 1. The motion sensor 20 may be installed integrally with the display unit 10. Further, it may be installed separately from the display unit 10.

  The motion sensor 20 includes an imaging unit 21 and an operation detection unit 22.

  The imaging unit 21 is one or a plurality of cameras that capture an image of a subject. The imaging unit 21 acquires a subject image that is a captured subject image. The subject image is preferably an image in which the whole body (whole body) of the subject is reflected, and includes a face image of the subject. The imaging unit 21 outputs the acquired subject image to the motion detection unit 22 and the face recognition unit 40 described later of the information device 30.

  The imaging unit 21 includes a sensor that measures a distance from the subject, such as a depth sensor, and can detect the position of the subject. The imaging unit 21 outputs the detected position information of the subject to the motion detection unit 22 and the face recognition unit 40 together with the acquired subject image.

  The motion detection unit 22 has a function of detecting the motion of the subject based on the subject image. The motion detection unit 22 may be configured by a motion analysis system motion analysis program or the like.

  The motion detection unit 22 uses the subject image acquired by the imaging unit 21 to acquire posture information that is information regarding the posture of the subject. The posture information includes information indicating the position of the posture feature point. Posture feature points are parts (nodes) that characterize the body shape and posture of the subject, such as the head, trunk, limbs, and joints. The motion detection unit 22 acquires posture information from the subject image acquired by the imaging unit 21 in real time, and outputs the posture information to the association unit 51 described later of the information device 30.

  For example, it is assumed that a captured image G including a subject image Ga of the subject A and a subject image Gb of the subject B is acquired by the imaging unit 21 as illustrated in FIG. In this case, the motion detection unit 22 analyzes the subject image Ga, and extracts posture feature points Pa1, Pa2,..., Pan (n pieces) of the subject A as shown in FIG. , Posture information Pa including these is acquired. The motion detection unit 22 analyzes the subject image Gb, extracts posture feature points Pb1, Pb2,..., Pbn (n pieces) of the subject B, and obtains posture information Pb including these. Note that n is a positive integer equal to or greater than 2, and is a number determined in advance according to the analysis capability of the motion detection unit 22. For example, n may be 25.

  As described above, the motion sensor 20 is configured so that the subject image acquired by the imaging unit 21 can be output to both the motion detection unit 22 and the face recognition unit 40. That is, the motion sensor 20 is configured such that the subject image used by the face recognition unit 40 and the subject image used by the motion detection unit 22 are acquired by one imaging unit 21. However, the image display device 1 is not limited to this, and the subject image used in the face recognition unit 40 and the subject image used in the motion detection unit 22 may be acquired by separate imaging units. Good. That is, the image display device 1 may be configured to include a dedicated face recognition camera that acquires a subject image used in the face recognition unit 40, separately from the motion sensor 20. Then, the image display device 1 outputs the subject image acquired by the face recognition camera to the face recognition unit 40, and outputs the subject image acquired by the imaging unit 21 of the motion sensor 20 to the motion detection unit 22. It may be configured.

  The information device 30 is a device that controls image display on the display unit 10 based on information output from the motion sensor 20 such as a subject image and posture information. The information device 30 may be configured with a personal computer.

  The information device 30 includes a face recognition unit 40 and a control unit 50.

  The face recognition unit 40 has a function of performing face recognition of a subject based on the subject image. The face recognition unit 40 may be configured by a face recognition application program executed by a processor (not shown) included in the information device 30.

  The face recognition unit 40 performs face recognition using a face image included in the subject image acquired by the imaging unit 21, and acquires attribute information that is information regarding the attribute of the subject. The attribute information is information such as the age and sex of the subject. The attribute information includes information indicating the position of the face feature point. The face feature points are parts characterizing the face of the subject such as eyes, nose, mouth, and eyebrows. The face recognition unit 40 outputs the acquired attribute information to the association unit 51 of the control unit 50.

  For example, as illustrated in FIG. 4A, it is assumed that a captured image G including a subject image Ga of the subject A and a subject image Gb of the subject B is acquired by the imaging unit 21. In this case, the face recognition unit 40 identifies the face image Fa included in the subject image Ga and the face image Fb included in the subject image Gb. Then, the face recognition unit 40 analyzes the face image Fa and extracts the face feature points Qa1, Qa2,..., Qam (m pieces) of the subject A, as shown in FIG. Attribute information Qa including these is acquired. Further, the face recognition unit 40 analyzes the face image Fb, extracts the face feature points Qb1, Qb2,..., Qbm (m pieces) of the subject B, and acquires attribute information Qb including these. Note that m is a positive integer equal to or greater than 2, and is a number determined in advance according to the analysis capability of the face recognition unit 40. For example, m may be 14.

  The control unit 50 has a function of creating an image model of a subject based on at least attribute information and causing the display unit 10 to display the image model. Specifically, the control unit 50 determines based on the attribute information whether the subject indicated in the subject image acquired by the imaging unit 21 is a display target of the display unit 10. When the control unit 50 determines that the subject is a display target of the display unit 10, the control unit 50 displays the image model created based on the attribute information on the display unit 10 according to the operation of the subject. Let An image model can be created based on the attribute information of a subject that is determined to be a display target, but is created based on both the posture information and attribute information of a subject that is determined to be a display target Is preferred. The control unit 50 may be configured by a three-dimensional CG (computer graphics) application program that is executed by a processor (not shown) included in the information device 30.

  The control unit 50 includes an association unit 51, a determination unit 52, a model creation unit 53, a costume determination unit 54, a model synthesis unit 55, a scene determination unit 56, and a display control unit 57.

  The associating unit 51 has a function of associating the posture information with the attribute information based on the position of the posture feature point included in the posture information and the position of the face feature point included in the attribute information. Note that the process of associating the posture information with the attribute information is also referred to as “association process”.

  The image acquired by the imaging unit 21 may include a plurality of subject images. In this case, the posture information acquired by the motion detection unit 22 may be composed of a plurality of posture information at the same time, and the attribute information acquired by the face recognition unit 40 may be composed of a plurality of posture information at the same time. In the control unit 50, the plurality of posture information output from the motion detection unit 22 and the plurality of attribute information output from the face recognition unit 40 have different data formats and are not associated with each other. May be entered. That is, the control unit 50 may be input in a state where the correspondence between the plurality of posture information output from the motion detection unit 22 and the plurality of attribute information output from the face recognition unit 40 is not clear. . For this reason, the associating unit 51 performs a process of associating the plurality of posture information output from the motion detecting unit 22 and the plurality of attribute information output from the face recognizing unit 40. Then, the associating unit 51 outputs the plurality of associated posture information and the plurality of attribute information to the determining unit 52.

  Details of the associating process performed using the associating unit 51 will be described later with reference to FIG.

  The determination unit 52 has a function of determining whether a subject corresponding to the attribute information is a display target of the display unit 10 using the attribute information. The process of determining whether or not there is a subject to be displayed on the display unit 10 and determining which subject is to be displayed is also referred to as “display target determination process”.

  As described above, the image acquired by the imaging unit 21 may include a plurality of subject images. In this case, the control unit 50 uses the determination unit 52 to set one or a plurality of subjects as display targets of the display unit 10 from among a plurality of subjects indicated in the plurality of subject images, and select an image model of the subject. It is displayed on the display unit 10. At this time, the determination unit 52 selects posture information and attribute information that meet a predetermined condition from the plurality of posture information and the plurality of attribute information associated by the associating unit 51. Then, the determination unit 52 determines that the subject corresponding to the selected posture information and attribute information is the display target of the display unit 10. Then, the determination unit 52 outputs the posture information and attribute information of the subject determined to be a display target to the model creation unit 53, the clothing determination unit 54, and the scene determination unit 56.

  The details of the display object determination process performed using the determination unit 52 will be described later with reference to FIG.

  The model creation unit 53 has a function of creating an image model of a subject to be displayed on the display unit 10. The image model created by the model creation unit 53 may be a three-dimensional CG model. If the subject is a person, the three-dimensional CG model created by the model creation unit 53 is a human body model that conforms to the body shape, posture, and attributes of the subject, and is added to the human body shape model and this shape model. Includes colors such as skin color. Note that the process of creating a three-dimensional CG model in the model creation unit 53 is also referred to as “model creation process”.

  The model creation unit 53 creates a three-dimensional CG model that matches the subject determined to be a display target by the determination unit 52 using the posture information and attribute information selected by the determination unit 52. That is, the model creation unit 53 creates a three-dimensional CG model according to the body shape, posture, and attributes of the subject determined as the display target, using the posture information and attribute information of the subject determined as the display target. Then, the model creation unit 53 outputs the created 3D CG model to the model synthesis unit 55.

  The clothing determination unit 54 has a function of determining a clothing model to be combined with the three-dimensional CG model created by the model creation unit 53. The clothing model is a three-dimensional CG model including a shape model of clothing such as clothes, shoes, and accessories, and patterns and colors added to the shape model. Note that the process of determining a clothing model is also referred to as “clothing determination processing”.

  A plurality of clothing models are stored in advance in a storage unit (not shown) included in the information device 30. The plurality of clothing models stored in the storage unit include, for example, clothing models related to recommended products, and are classified according to date, time zone, season, event, location, plan, and the like. The clothing determination unit 54 selects a clothing model that matches a predetermined clothing determination condition from a plurality of clothing models stored in the storage unit, and combines the clothing model with the three-dimensional CG model created by the model creation unit 53. Decide on a model. This predetermined clothing determination condition is set based on the attribute information of the subject determined to be a display target. Then, the clothing determination unit 54 outputs the determined clothing model to the model synthesis unit 55 and the scene determination unit 56.

  The model synthesis unit 55 has a function of synthesizing the clothing model determined by the clothing determination unit 54 with the three-dimensional CG model created by the model creation unit 53. The model synthesizing unit 55 synthesizes the clothing model determined by the clothing determining unit 54 according to the three-dimensional CG model created by the model creating unit 53. Then, the model synthesis unit 55 outputs the three-dimensional CG model in which the clothing model is synthesized to the display control unit 57. The process of synthesizing the clothing model with the three-dimensional CG model created by the model creation unit 53 is also referred to as “model synthesis process”.

  The scene determination unit 56 has a function of determining scene information to be displayed on the display unit 10. The scene information is space information that can form a virtual three-dimensional space in which the three-dimensional CG model synthesized by the model synthesis unit 55 is arranged. The scene information includes a background image and a scene object. The scene object is an object other than the three-dimensional CG model synthesized by the model synthesis unit 55. The process for determining scene information is also referred to as “scene determination process”.

  A plurality of scene information is stored in advance in a storage unit included in the information device 30. The plurality of pieces of scene information stored in the storage unit are classified according to, for example, date, time zone, season, event, location, planning theme, etc., and are stored in association with a plurality of clothing models. The scene determination unit 56 selects scene information corresponding to the clothing model determined by the clothing determination unit 54 from the plurality of scene information stored in the storage unit, and determines the scene information to be displayed on the display unit 10. . Then, the scene determination unit 56 outputs the determined scene information to the display control unit 57.

  The display control unit 57 has a function of generating a display image of the display unit 10 and performing display control on the display unit 10. That is, the display control unit 57 performs rendering using the three-dimensional CG model synthesized by the model synthesis unit 55 and the scene information determined by the scene determination unit 56 to generate a display image of the display unit 10. . Then, the display control unit 57 outputs an image signal indicating the generated display image to the display unit 10 and causes the display unit 10 to display the generated display image. The process of generating a display image of the display unit 10 and performing display control on the display unit 10 is also referred to as “display control process”.

  The display control unit 57 causes the display unit 10 to display the image model of the subject determined as the display target in accordance with the operation of the subject. Specifically, the display control unit 57 receives the posture information of the subject determined to be a display target from the determination unit 52, and is synthesized by the model synthesis unit 55 in accordance with the change of the posture information according to the motion of the subject. The three-dimensional CG model is moved and displayed on the display unit 10. Thereby, the image of the subject and the three-dimensional CG model of the clothing displayed on the display unit 10 moves following the movement of the subject determined as the display target.

  Note that the information device 30 may include a communication unit capable of bidirectional communication with a server device that manages the image display device 1 and another image display device 1. For example, the information device 30 can download the clothing model, the scene information, the application program, and the like from the server device using the communication unit, and store them in the storage unit of the information device 30. Thereby, the information device 30 can appropriately update the clothing model, the scene information, various application programs, and the like.

[2. Detailed flow of various processes]
FIG. 5 is a flowchart showing a detailed flow of image display processing performed in the image display apparatus 1 according to the embodiment of the present invention. FIG. 6 is a diagram showing a display image displayed on the image display device 1 according to the embodiment of the present invention. FIG. 6A shows a demo image. FIG. 6B shows a three-dimensional CG model image of the subject displayed based on the posture information and the attribute information. FIG. 6C shows an image of a three-dimensional CG model that moves according to the movement of the subject. FIG. 6D shows an image in which the clothing model is changed.

  The image display process shown in FIG. 5 may be a process performed at a predetermined cycle after the image display device 1 is activated.

  In step S <b> 501, the image display device 1 displays a demonstration image on the display unit 10. The demo image is an image displayed on the display unit 10 when the subject does not exist within the angle of view of the imaging unit 21. As shown in FIG. 6A, the demo image may be, for example, a still image of a human body model. The image of the human body model shown in FIG. 6A may be an image representing the mannequin with a three-dimensional CG model. The image of the clothing model shown in FIG. 6A may be, for example, an image of the clothing model related to the recommended product.

  In step S <b> 502, when the subject enters the angle of view of the imaging unit 21, the image display apparatus 1 acquires a subject image using the imaging unit 21. The subject image is shown as shown in FIG.

  In step S503, the image display device 1 uses the motion detection unit 22 to detect the motion of the subject based on the subject image, and acquires posture information including posture feature points. In the example of FIGS. 3A and 3B, the image display device 1 acquires the posture information Pa of the subject image Ga and the posture information Pb of the subject image Gb. The posture information Pa includes posture feature points Pa1 to Pan. The posture information Pb includes posture feature points Pb1 to Pbn.

  In step S504, the image display device 1 performs face recognition from the subject image using the face recognition unit 40, and acquires attribute information including face feature points. 4A and 4B, the image display apparatus 1 acquires the attribute information Qa of the subject A from the face image Fa included in the subject image Ga, and at the same time the face image included in the subject image Gb. The attribute information Qb of the subject B is acquired from Fb. The attribute information Qa includes face feature points Qa1 to Qam. The attribute information Qb includes face feature points Qb1 to Qbm.

  In step S <b> 505, the image display apparatus 1 performs association processing using the association unit 51. By executing the association process, the image display device 1 can clarify the correspondence between the posture information and the attribute information and associate the posture information and the attribute information with each other.

  The detailed flow of the association process will be described later with reference to FIG.

  In step S <b> 506, the image display device 1 uses the determination unit 52 to perform display target determination processing. By executing the display target determination process, the image display apparatus 1 determines whether or not there is a subject to be displayed, determines which subject is the display target, and posture information and attributes of the subject determined to be the display target. Information can be selected.

  The detailed flow of the display target determination process will be described later with reference to FIG.

  In step S507, the image display apparatus 1 determines whether it is determined by the display target determination process that there is no subject to be displayed. If it is not determined that there is no subject to be displayed, the image display device 1 proceeds to step S508. On the other hand, when it is determined that there is no subject to be displayed, the image display device 1 ends this process.

  In step S <b> 508, the image display apparatus 1 performs model creation processing using the model creation unit 53. The image display device 1 uses the posture information and attribute information selected by the display target determination process to create a three-dimensional CG model that conforms to the body shape, posture, and attributes of the subject determined to be a display target.

  In step S <b> 509, the image display apparatus 1 performs a clothing determination process using the clothing determination unit 54. The image display device 1 is a clothing model that conforms to a predetermined clothing determination condition set based on attribute information selected by the display target determination process from among a plurality of clothing models stored in the storage unit of the information device 30. Is selected. Then, the image display device 1 determines the selected clothing model as a clothing model to be combined with the three-dimensional CG model created by the model creation processing.

  In step S <b> 510, the image display device 1 performs model synthesis processing using the model synthesis unit 55. The image display device 1 synthesizes the clothing model determined by the clothing determination processing in accordance with the three-dimensional CG model created by the model creation processing.

  In step S <b> 511, the image display apparatus 1 performs a scene determination process using the scene determination unit 56. The image display device 1 selects scene information corresponding to the clothing model determined by the clothing determination process from the plurality of scene information stored in the storage unit of the information device 30. Then, the image display device 1 determines the selected scene information as scene information to be displayed together with the three-dimensional CG model synthesized by the model synthesis process.

  In step S <b> 512, the image display apparatus 1 performs display control processing using the display control unit 57. The image display device 1 performs rendering using the three-dimensional CG model synthesized by the model synthesis process and the scene information determined by the scene determination process, and generates a display image of the display unit 10. Then, the image display device 1 outputs an image signal indicating the generated display image to the display unit 10 and causes the display unit 10 to display the generated display image.

  For example, as shown in FIG. 6B, it is assumed that the subject determined to be a display target is subject A, which is a slim male in the late 20s. Then, the posture of the subject A determined to be a display target is a posture standing with a hand on the hip. Assume that the image display device 1 is in a location where a person interested in soccer such as the vicinity of a soccer field can easily pass. Then, in the clothing determination process, it is assumed that a clothing model related to soccer training wear is determined as a recommended product for a slim-type man in his late 20s. In this case, as shown in FIG. 6 (b), the display unit 10 has a three-dimensional CG model that conforms to the body shape, posture, and attributes of the subject A, and clothing related to soccer training wear that conforms to the subject attributes. The combined image is displayed. For example, the display unit 10 displays an image of scene information indicating a soccer field corresponding to the determined clothing model.

  In step S513, the image display device 1 determines whether or not the posture of the subject determined to be a display target has changed. Whether the image display device 1 has changed the posture of the subject determined to be the display target by checking whether the position of the posture feature point has been changed in the posture information of the subject determined to be the display target. It may be determined whether or not. When the posture of the subject has not changed, the image display device 1 proceeds to step S512, and continuously displays the image displayed on the display unit 10 on the display unit 10. On the other hand, when the posture of the subject has changed, the image display device 1 proceeds to step S514.

  In step S514, the image display device 1 determines whether or not to end the display on the display unit 10. The image display device 1 determines whether or not the motion of the subject determined as the display target is a predetermined display end motion. The display end operation may be, for example, an operation of leaving from the front of the image display device 1. The image display device 1 confirms whether or not the position of the posture feature point has been changed so as to indicate the display end operation included in the posture information of the subject determined to be a display target. It may be determined whether or not to end. The change of the position of the posture feature point so as to indicate the display end operation may be, for example, that the position of the posture feature point changes at a certain speed or more over a certain time. When the display on the display unit 10 is ended, the image display device 1 ends the present process. On the other hand, when the display on the display unit 10 is not terminated, the image display device 1 proceeds to step S515.

  In step S515, the image display device 1 determines whether to change the clothing model displayed on the display unit 10. The image display device 1 determines whether or not the motion of the subject that is determined as a display target is a predetermined clothing model changing operation. The change operation of the clothing model may be, for example, an operation of taking off clothes. The image display device 1 confirms whether or not the position of the posture feature point is included in the posture information of the subject determined to be a display target so as to indicate the change operation of the clothing model, thereby displaying the display unit 10. It may be determined whether to change the clothing model displayed on the screen.

  When changing the clothing model, the image display device 1 proceeds to step S509 and performs the clothing determination process again. At that time, the image display device 1 corrects the clothing determination condition so that a clothing model different from the currently displayed clothing model is selected.

On the other hand, when the clothing model is not changed, the image display device 1 proceeds to step S512 and performs display control processing again. At that time, the image display device 1 moves the three-dimensional CG model synthesized by the model synthesis unit 55 in accordance with the change in the position of the posture feature point included in the posture information of the subject determined as the display target. Display control processing is performed.

  For example, as shown in FIG. 6 (c), a subject A determined to be a display target is placed with a soccer ball from a posture standing with his / her hand on his / her waist as shown in FIG. 6 (b). Suppose that an action that changes to a kicking posture is performed. It is assumed that this operation does not correspond to the display end operation or the clothing model change operation. In this case, as shown in FIG. 6 (c), the display unit 10 has the soccer training wear shown in FIG. Then, an image of a three-dimensional CG model that operates so as to change to a posture of kicking a soccer ball is displayed.

  Further, for example, as shown in FIG. 6 (d), the subject A determined to be a display object is in a posture standing with his / her hand on the waist as shown in FIG. 6 (b). Suppose you perform an action that changes your posture to take off. It is assumed that this operation corresponds to a clothing model change operation. In this case, for example, as shown in FIG. 6D, the display unit 10 has an image of a three-dimensional CG model that has been changed to training wear of a color different from the training wear shown in FIG. Is displayed.

  Thereafter, for example, when the subject A determined to be displayed performs the above-described display end operation, a demo image as shown in FIG. 6A is displayed on the display unit 10.

  FIG. 7 is a flowchart showing a detailed flow of the associating process performed in the image display apparatus 1 according to the embodiment of the present invention.

  When there is a single subject, the correspondence relationship between the acquired posture information and the attribute information is obvious as a one-to-one relationship, and thus is naturally associated. On the other hand, when there are a plurality of subjects, the correspondence between the acquired plurality of posture information and the plurality of attribute information is not clear. Therefore, in the description of FIG. 7 and FIG. 8, a case where the subject is two people, the subject A and the subject B, will be described as an example.

  In step S701, the posture information Pa and Pb acquired in step S503 in FIG. 5 and the attribute information Qa and Qb acquired in step S504 in FIG. 5 are input to the associating unit 51 of the image display device 1. .

  In step S702-1, the image display apparatus 1 determines the positions of the posture feature points Pa1 to Pan included in the posture information Pa, the positions of the posture feature points Pb1 to Pbn included in the posture information Pb, and these position coordinate systems XP. Confirm.

  In step S702-2, the image display apparatus 1 determines the positions of the face feature points Qa1 to Qam included in the attribute information Qa, the positions of the face feature points Qb1 to Qbm included in the attribute information Qb, and the position coordinate system XQ. Confirm. Note that step S702-1 and step S702-2 may be processed in parallel.

  In step S <b> 703, the image display apparatus 1 performs coordinate conversion of the position coordinate system XP of the posture feature points and the position coordinate system XQ of the face feature points into the common coordinate system X. It is assumed that the posture feature points Pa1 to Pan expressed in the position coordinate system XP are converted into posture feature points Pa1 'to Pan' by coordinate conversion to the common coordinate system X, respectively. Similarly, it is assumed that the posture feature points Pb1 to Pbn represented in the position coordinate system XP are converted into posture feature points Pb1 'to Pbn' by coordinate conversion to the common coordinate system X, respectively. Further, it is assumed that the face feature points Qa1 to Qam expressed in the position coordinate system XQ are converted into face feature points Qa1 'to Qam' by the coordinate conversion to the common coordinate system X, respectively. Similarly, it is assumed that the face feature points Qb1 to Qbm expressed in the position coordinate system XQ are converted into face feature points Qb1 'to Qbm' by coordinate conversion to the common coordinate system X, respectively.

  In step S704-1, the image display apparatus 1 confirms the positions of the posture feature points Pa1 'to Pan' and the positions of the posture feature points Pb1 'to Pbn' in the common coordinate system X.

  In step S704-2, the image display apparatus 1 checks the positions of the face feature points Qa1 'to Qam' and the positions of the face feature points Qb1 'to Qbm' in the common coordinate system X. Note that step S704-1 and step S704-2 may be processed in parallel.

  In step S705, the image display device 1 determines the average position of the posture feature points Pa1 ′ to Pan ′, the average position of the posture feature points Pb1 ′ to Pbn ′, the average position of the face feature points Qa1 ′ to Qam ′, and the face feature. The average position of the points Qb1 ′ to Qbm ′ is calculated.

  In step S706, the image display apparatus 1 determines the average position of the posture feature points Pa1 ′ to Pan ′, the average position of the posture feature points Pb1 ′ to Pbn ′, the average position of the face feature points Qa1 ′ to Qam ′, and The face feature points Qb1 ′ to Qbm ′ are collated with the average position.

  For example, the image display device 1 calculates the distance between the average position of the posture feature points Pa1 'to Pan' and the average position of the face feature points Qa1 'to Qam'. The image display device 1 calculates the distance between the average position of the posture feature points Pa1 'to Pan' and the average position of the face feature points Qb1 'to Qbm'. Similarly, the image display device 1 calculates the distance between the average position of the posture feature points Pb1 'to Pbn' and the average position of the face feature points Qa1 'to Qam'. The image display device 1 calculates the distance between the average position of the posture feature points Pb1 'to Pbn' and the average position of the face feature points Qb1 'to Qbm'. Then, the image display device 1 obtains a combination having the shortest distance between the average position of the posture feature points and the average position of the face feature points. In the combination of the posture feature point and the face feature point with the shortest distance, the average position of the posture feature point and the average position of the face feature point are the closest. For this reason, the image display apparatus 1 estimates that the combination of the posture feature point and the face feature point with the shortest distance is a combination in which the feature point is extracted from the same subject image.

  Here, it is assumed that the average position of the posture feature points Pa1 'to Pan' is closer to the average position of the face feature points Qa1 'to Qam' than the average position of the face feature points Qb1 'to Qbm'. Assume that the average positions of the posture feature points Pb1 'to Pbn' are closer to the average positions of the face feature points Qb1 'to Qbm' than the average positions of the face feature points Qa1 'to Qam'.

  In step S707-1, the image display apparatus 1 includes posture information Pa including posture feature points Pa1 to Pan corresponding to posture feature points Pa1 ′ to Pan ′ and face feature points Qa1 ′ to Qam that are close to each other in average position. The attribute information Qa including the face feature points Qa1 to Qam corresponding to 'is associated.

  In step S707-2, the image display apparatus 1 includes posture information Pb including posture feature points Pb1 to Pbn corresponding to posture feature points Pb1 ′ to Pbn ′ and face feature points Qb1 ′ to Qbm that are close to each other in average position. The attribute information Qb including the face feature points Qb1 to Qbm corresponding to 'is associated. Step S707-1 and step S707-2 may be processed in parallel.

  In step S708-1, the image display device 1 stores the associated posture information Pa and attribute information Qa as subject information Ta of the subject A.

  In step S708-2, the image display device 1 stores the associated posture information Pb and attribute information Qb as subject information Tb of the subject B. Note that step S708-1 and step S708-2 may be processed in parallel. Thereafter, the image display device 1 ends the process, and proceeds to step S506 in FIG.

  Note that the image display device 1 calculates the average position of the posture feature points and the average position of the face feature points in step S705, and collates the average position of the posture feature points and the average position of the face feature points in step S706. However, the image display device 1 is not limited to this. In step S705, the representative position of the posture feature point and the representative position of the face feature point are specified. In step S706, the representative position of the posture feature point and the representative position of the face feature point are determined. May be matched. The representative position of the posture feature point may be, for example, the position of the posture feature point indicating the head. The representative position of the face feature point may be, for example, the position of the face feature point indicating the nose.

  FIG. 8 is a flowchart showing a detailed flow of the display target determination process performed in the image display apparatus 1 according to the embodiment of the present invention.

  In step S801, the image display apparatus 1 confirms subject information Ta of the subject A including the posture information Pa and the attribute information Qa, and subject information Tb of the subject B including the posture information Pb and the attribute information Qb.

  In step S <b> 802, the image display device 1 searches for subject information including attribute information that satisfies the condition 1. Condition 1 is one of the display target conditions indicating the conditions under which the subject can be a display target, and may be a condition related to attribute information. For example, the condition 1 may be set such that the age included in the subject attribute information is 30 years old or lower. If the information about the age of the attribute information Qa is in the late 20s and the information about the age of the attribute information Qb is in the early 20s, the search result according to the condition 1 includes subject information Ta including the attribute information Qa. And subject information Tb including attribute information Qb.

  In step S <b> 803, the image display device 1 determines whether or not subject information including attribute information that satisfies the condition 1 is present in the search result according to the condition 1. If there is no subject information including attribute information that satisfies condition 1, the image display apparatus 1 proceeds to step S808. On the other hand, when there is subject information including attribute information that satisfies the condition 1, the image display device 1 proceeds to step S804.

  In step S <b> 804, the image display device 1 searches the subject information including the attribute information matching the condition 2 from the search results related to the condition 1. Similarly to condition 1, condition 2 is one of the display target conditions indicating the conditions under which the subject can be a display target, and may be a condition related to attribute information. For example, the condition 2 may be set so that the gender included in the subject attribute information is male. If the information regarding the gender of the attribute information Qa is male and the information regarding the gender of the attribute information Qb is female, the search result according to the condition 2 includes only subject information Ta including the attribute information Qa.

  In step S <b> 805, the image display apparatus 1 determines whether or not subject information including attribute information that satisfies the condition 2 is present in the search result according to the condition 2. If there is no subject information including attribute information that meets condition 2, the image display device 1 proceeds to step S808. On the other hand, when there is subject information including attribute information that satisfies the condition 2, the image display device 1 proceeds to step S806.

  In step S <b> 806, the image display device 1 searches subject information including posture information that satisfies the condition 3 from the search results according to the condition 2. Condition 3 is one of the display target conditions indicating the conditions under which the subject can be a display target, similar to Condition 1 and Condition 2. However, unlike Condition 1 and Condition 2, the condition 3 is a condition related to posture information. Also good. For example, the condition 3 may be set so that the position of the posture feature point included in the posture information of the subject does not change or changes at a predetermined speed or less. In other words, the condition 3 may be set such that the moving speed of the subject is not less than zero and not more than a predetermined speed. The condition 3 is a condition for setting a subject that is interested in the image display device 1 as a display target, such as stopping in front of the image display device 1 or passing while paying attention to the image display device 1. If the subject A is stopped in front of the image display device 1, the speed at which the position of the posture feature points Pa <b> 1 to Pan included in the posture information Pa changes is at least a predetermined speed or less. This includes subject information Ta including posture information Pa.

  In step S807, the image display device 1 selects a subject corresponding to the subject information that satisfies the condition 3 as a subject to be displayed. The image display device 1 selects the subject A as the subject to be displayed because the posture information Pa included in the subject information Ta of the subject A meets the condition 3. Thereafter, the image display device 1 ends the process, and proceeds to step S507 in FIG.

  In step S808, the image display device 1 determines that there is no subject to be displayed. Then, the image display device 1 stores that it is determined that there is no subject to be displayed, and then ends this processing, and proceeds to step S507 in FIG.

  Note that the image display apparatus 1 searches in the order of the above-described conditions 1 to 3 when searching for subject information that matches a display target condition indicating a condition that the subject can be a display target. However, the image display device 1 is not limited to this. For example, the image display device 1 may first be searched with the condition 3 related to the posture information. Further, the number of display target conditions is not limited to the above conditions 1 to 3. Further, the display target conditions may all be conditions related to the attribute information, and the contents thereof are not limited to the above conditions 1 to 3.

[3. Effect]
As described above, the image display apparatus 1 according to the present embodiment captures a subject and obtains a subject image, and uses the subject image to detect the motion of the subject in real time to obtain posture information. And an operation detection unit 22. For this reason, the image display apparatus 1 according to the present embodiment does not need to measure and store the three-dimensional human body shape data and posture data of the subject in advance, and performs calculations to estimate the posture and motion of the subject. The amount does not become enormous. Therefore, the image display apparatus 1 according to the present embodiment can accurately represent the posture and motion of the subject to be displayed even with a simple configuration, so that the cost can be reduced and easy introduction is possible. It becomes.

  In addition, the image display device 1 according to the present embodiment displays a 3D CG model image of the subject instead of combining and displaying the clothing model on the subject image acquired by the imaging unit 21. For this reason, in the image display device 1 according to the present embodiment, in the display image of the display unit 10, the image of the clothing item to be tried on does not interfere with the image of the clothing item currently worn by the subject. Moreover, the image display device 1 according to the present embodiment can create a three-dimensional CG model of the subject based on the posture information and the attribute information and display it according to the posture and motion of the subject. Moreover, in the image display device 1 according to the present embodiment, a clothing model determined based on the attribute information of the subject is combined with the three-dimensional CG model of the subject. For this reason, the image display apparatus 1 according to the present embodiment can display the images of the 3D CG model and the clothing model in accordance with the body shape, posture, and attributes of the subject according to the posture and motion of the subject. . Therefore, the image display apparatus 1 according to the present embodiment can display a try-on image that is truly suitable for the subject to be displayed.

  Furthermore, the image display device 1 according to the present embodiment includes an associating unit 51 that associates posture information and attribute information. For this reason, in the image display device 1 according to the present embodiment, the motion detection unit 22 that acquires posture information and the face recognition unit 40 that acquires attribute information are separately developed, and the data format of posture information and attribute information Even if they are different, a three-dimensional CG model can be created by associating the two. Thereby, the image display device 1 according to the present embodiment can easily create a three-dimensional CG model that conforms to the body shape, posture, and attributes of a subject even if a commercially available motion sensor that acquires posture information is introduced as it is, for example. can do. Therefore, in the image display device 1 according to the present embodiment, an image display device that can accurately represent the posture and motion of the subject to be displayed and can display a try-on image that is truly suitable for the subject to be displayed has a simple configuration. This makes it easy to implement.

  Furthermore, the image display apparatus 1 according to the present embodiment appropriately associates the plurality of subjects even when the plurality of subjects are captured and the plurality of posture information and the plurality of attribute information are acquired, and determines the determination unit 52. The object to be displayed can be selected intentionally and displayed. Therefore, the image display apparatus 1 according to the present embodiment can be used even in a situation where a plurality of subjects are assumed to be imaged simultaneously. In particular, the image display device 1 according to the present embodiment displays not only a substitute for a conventional fitting room but also a fitting image that is truly suitable for a subject to be displayed in a show window where a plurality of passersby come and go. New usage forms can be provided.

  Further, the image display device 1 according to the present embodiment displays a three-dimensional CG model image of the mannequin with the clothing recommended by the provider of the image display device 1 as a demo image in a situation where the subject is not imaged by the imaging unit 21. Can be made. The image display device 1 according to the present embodiment recommends a clothing model of clothing suitable for the subject in a three-dimensional manner of the subject in a situation where the subject is imaged by the imaging unit 21 and recommended by the provider of the image display device 1. It can be combined with a CG model and displayed. For this reason, the image display device 1 according to the present embodiment advertises clothing accessories to passers-by as well as a virtual fitting device that displays an image in which a subject to be displayed virtually tries on clothing. It can also serve as a digital signage device. Therefore, the image display device 1 according to the present embodiment can provide a new application of an image display device that serves as both a virtual fitting device and a digital signage device.

  In particular, the image display device 1 according to the present embodiment displays a still image of a mannequin on which clothing items are worn while displaying a demo image, and when the subject to be displayed appears, this subject An image of a three-dimensional CG model that moves according to the posture and motion of the user can be displayed. For this reason, the image display apparatus 1 according to the present embodiment can make a passerby including a subject to be displayed feel that the image of the three-dimensional CG model displayed on the display unit 10 suddenly starts to move. Therefore, the image display device 1 according to the present embodiment can function as an advertising medium having a higher advertising effect than the conventional digital signage device.

[4. Others]
In the above-described embodiment, the image display device 1 corresponds to an example of an “image display device” recited in the claims. The display unit 10 corresponds to an example of a “display unit” recited in the claims. The imaging unit 21 corresponds to an example of an “imaging unit” recited in the claims. The motion detection unit 22 corresponds to an example of an “motion detection unit” recited in the claims. The face recognition unit 40 corresponds to an example of a “face recognition unit” recited in the claims. The control unit 50 corresponds to an example of a “control unit” recited in the claims. The model creation unit 53 corresponds to an example of a “model creation unit” described in the claims. The display control unit 57 corresponds to an example of a “display control unit” recited in the claims. The associating unit 51 corresponds to an example of an “associating unit” described in the claims.

  It will be apparent to those skilled in the art that the above-described embodiments can be applied to each other, including modifications.

  The above description is intended to be illustrative rather than limiting. Thus, it will be apparent to one skilled in the art that modifications may be made to the embodiments of the present invention without departing from the scope of the claims.

  Terms used throughout the specification and claims are to be interpreted as “non-limiting” terms. For example, the terms “include” and “included” should be interpreted as “not limited to those described as included”. The term “comprising” should be interpreted as “not limited to what is described as having”.

DESCRIPTION OF SYMBOLS 1 Image display apparatus 10 Display part 20 Motion sensor 21 Image pick-up part 22 Motion detection part 30 Information equipment 40 Face recognition part 50 Control part 51 Association part 52 Judgment part 53 Model creation part 54 Costume determination part 55 Model composition part 56 Scene determination part 57 Display control unit A Subject B Subject Fa Facial image Fb Facial image G Captured image Ga Subject image Gb Subject image Pa Posture information Pa1-Pan Posture feature point Pb Posture information Pb1-Pbn Posture feature point Qa Attribute information Qa1-Qam Facial feature point Qb Attribute information Qb1 to Qbm Face feature points Ta Subject information Tb Subject information

Claims (4)

A display for displaying an image;
An imaging unit that images a subject and obtains a subject image;
A motion detector that detects motion of the subject using the subject image and acquires posture information of the subject;
A face recognition unit that performs face recognition of the subject using the subject image and acquires attribute information of the subject;
A control unit that creates an image model of the subject and displays the image model on the display unit;
With
The controller is
Determining whether the subject is a display target of the display unit based on the attribute information;
When it is determined that the subject is a display target of the display unit, the image model created based on the attribute information is displayed on the display unit according to the motion of the subject based on the posture information.
Image display device. The controller is
A model creation unit that creates a three-dimensional CG model as the image model using the attribute information;
A display control unit that moves the three-dimensional CG model in accordance with a change in the posture information according to the movement of the subject and displays the display unit on the display unit;
The image display device according to claim 1, comprising: The motion detection unit detects the position of the posture feature point of the subject,
The face recognition unit detects a position of a face feature point of the subject;
The control unit further includes an association unit that associates the posture information and the attribute information based on the position of the posture feature point and the position of the face feature point,
The model creation unit creates the three-dimensional CG model using the posture information and the attribute information associated by the association unit.
The image display device according to claim 2. The association unit includes
The coordinate system of the position of the posture feature point and the coordinate system of the position of the face feature point are respectively converted into a common coordinate system,
In the common coordinate system, by comparing the position of the posture feature point and the position of the face feature point, the posture information and the attribute information are associated with each other.
The image display device according to claim 3.