JP2005142975A - Audience reaction information collection system and its method, user terminal and audience reaction information providing apparatus used in the audience reaction information collection system, and audience reaction information creation program used to obtain the user terminal and the audience reaction information providing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the load of work such as entry by an audience himself/ herself, and to provide a technique for automatically summing up the quality-of-view of contents in an ordinary viewing style. <P>SOLUTION: In a sensing apparatus, audience sensing data or audience recognition data resulting from recognizing the sensing data are collected to collect audience reaction data indicating the reaction of the audience with respect to reproduced contents. On the other hand, when viewing data are collected with respect to reproduced contents including screen operation history information in a user terminal, the user terminal or the information providing apparatus temporally synchronizes and integrates the collected audience reaction data and the collected viewing data. Further, the information providing apparatus generates quality-of-view data indicating the qualitative reaction of the audience to the reproduced contents on the basis of the integrated data, and viewing meta-data are created for distributing the generated quality-of-view data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a viewer reaction information collection system and method for measuring and providing “quality” of viewing “how content (such as a TV program) is viewed” that cannot be understood only by the audience rating and access log, and the method thereof A user terminal used in the viewer reaction information collecting system, a viewer reaction information providing device used in the viewer reaction information collecting system, and a program for creating viewer reaction information used in realizing the user terminal, Further, the present invention relates to a program for creating viewer response information used for realizing the viewer response information providing apparatus.

  Until now, as an index to quantitatively evaluate programs and content (measure the value), not only the “rate” that represents the “amount” of viewing but also the quality of viewing (hereinafter referred to as “viewing quality”) It has been screamed that a scale is necessary. In order for each broadcaster and content distributor to understand the effectiveness of the advertisement and explain it to the advertiser, the viewing quality is expected to become an increasingly important index in the future.

  In addition, terrestrial digital broadcasting is scheduled to start in December 2003 in the three major metropolitan areas, and the number of channels that can be viewed will increase to several hundred units due to the increase in the number of channels. Nevertheless, it is very difficult for the viewer to select from hundreds of channels for a limited viewing time of about 3 hours a day. In order to provide program navigation services and automatic program recommendation services that reduce the burden of program selection, it is necessary to focus on "how viewers watch TV programs" that have been neglected so far. .

  The present invention relates to a method of surveying content audience quality. In particular, by focusing attention on the behavior and reaction of viewers, it is possible to determine whether or not a person is actually watching a TV program or digital content, and It is possible to estimate the degree of interest of how much people are interested / interested.

  The content in the present invention refers to a television broadcast program, digital content on the Internet, an advertisement, or a combination thereof. Furthermore, for example, the estimation of the number of viewers using an IC tag (RFID tag, LED tag) or the like and personal identification of the viewer are realized at high speed and with high accuracy.

  As one of the conventional audience rating measurement techniques, there is an apparatus for grasping the number of viewers or specifying individual viewers using infrared video (see, for example, Patent Document 1).

  This device enables the determination of the number of viewers and whether a person is actually watching television by performing human pupil detection using an infrared video camera. Furthermore, by using the position information of the pupil to obtain an image having a constant two-dimensional position and size of the face in the video camera image output signal, it is possible to quickly and easily identify an individual viewer from the image of the face. It is.

  On the other hand, there is a web site “TV Program Audience Quality Survey Research Q” that conducts surveys on TV programs using the Internet, which started in April 1997 (see Non-Patent Document 1, for example).

This site was developed and operated in collaboration with the TV Asahi Organization Department and the Keio University Faculty of Environment and Information Sciences Kumazaka Lab. Has been. In addition, there is an incentive for users who have registered as members to collect presents by lottery by answering a questionnaire and collecting Q points.
Japanese Patent No. 2947289 “TV Television Rating System” "TV Program Audience Quality Research Q", Internet <URL: http://www.rq-tv.com>) 2003.8.25

  However, the conventional technique for measuring the audience rating cannot cover the content audience quality survey aimed at by the present invention. For example, it is possible to improve the accuracy of the so-called audience rating by grasping the situation where the viewer is not watching the television while the television is turned on. However, audience quality is not good.

  On the other hand, in the above-mentioned “Research Q”, depending on the contents of the questionnaire and the question items, it is considered that the audience quality can be investigated. However, considering the hassle of starting up a PC, connecting to the Internet, then answering a questionnaire on the Web site, or accessing a mobile phone-only Web site and entering a comment, It is hard to say that there are few.

  In view of the above-mentioned points, the present invention solves the problems of the prior art, reduces the work load such as entry by the viewer himself / herself and answers on the web, and automatically calculates the audience quality of the content in the normal viewing mode. The purpose is to provide a new content audience quality survey technology.

[1] Configuration in which viewing response data and viewing data are integrated in a user terminal In order to achieve the above-described object, a viewer response information collecting system of the present invention includes (a) sensing data about a viewer and A sensing device that collects viewer response data indicating viewer response to the playback content by collecting either or both of the viewer recognition data obtained by the recognition processing; ) Playing back content, collecting viewing data on the playback content including screen operation history information, obtaining viewer response data from the sensing device, and collecting the collected viewing data and the obtained viewer response User terminal that integrates data in time synchronization and (c) Integrated data obtained from the user terminal and the obtained integrated data A viewer response information providing device for generating audience quality data indicating a viewer's qualitative reaction to the playback content and creating viewing metadata for distribution of the generated audience quality data, Is provided.

[1-1] Configuration of user terminal in accordance with this configuration In accordance with this configuration, the user terminal (a) means for reproducing the content requested by the viewing request transmitted from the content providing device, and (b) the content providing device. (C) means for acquiring screen operation history information by obtaining screen operation information obtained by the viewer and storing it in the storage means; ) A means for obtaining viewer response data from the sensing device, and (e) integrating the collected screen operation history information and the obtained viewer response data in a time-synchronized manner, and reproducing content metadata. And (f) means for transmitting the integrated data to the viewer response information providing apparatus.

  Each of the processing means described above can also be realized by a computer program, and this computer program can be provided by being recorded on a recording medium such as a semiconductor memory or via a network.

[1-2] Configuration of Viewer Response Information Providing Device According to This Configuration When this configuration is followed, the viewer response information providing device includes: (b) means for obtaining integrated data from the user terminal; Based on the obtained integrated data, means for generating audience quality data indicating the viewer's qualitative reaction to the playback content, and (c) creating viewing metadata for distribution of the generated audience quality data And (d) means for transmitting the created viewing metadata to the request source.

  Each of the above processing means can also be realized by a computer program, and this computer program can be provided by being recorded in a recording medium such as a semiconductor memory or via a network.

[2] Configuration for Integrating Viewing Response Data and Viewing Data in Viewer Response Information Providing Device In order to achieve the above-described object, the viewer response information collecting system of the present invention provides (a) sensing for a viewer. Sensing device for collecting viewer response data indicating viewer's response to playback content by collecting either or both of data and recognition data of viewer obtained by recognizing the data And (b) a user terminal that reproduces the content and collects viewing data regarding the reproduced content including information on the screen operation history, and (c) obtains the viewing data from the user terminal and receives a viewer response from the sensing device. The data is obtained, and the obtained viewing data and the obtained audience reaction data are integrated in time synchronization. Viewer response information is provided to generate audience quality data indicating the viewer's qualitative reaction to the playback content based on the recorded data and to create viewing metadata for distribution of the generated audience quality data Device.

[2-1] Configuration of user terminal in accordance with this configuration In the case of following this configuration, the user terminal (a) means for reproducing the content requested for viewing transmitted from the content providing device, and (b) the content providing device. (C) means for acquiring screen operation history information by obtaining screen operation information obtained by the viewer and storing it in the storage means; ) Means for creating viewing data by processing the collected screen operation history information and the metadata of the obtained reproduction content; and (e) transmitting the created viewing data to the viewer response information providing apparatus. And means.

  Each of the above processing means can also be realized by a computer program, and this computer program can be provided by being recorded in a recording medium such as a semiconductor memory or via a network.

[2-2] Configuration of Viewer Response Information Providing Device in Case of Following this Configuration In the case of following this configuration, the viewer response information providing device includes: (b) means for obtaining viewer response data from the sensing device; ) Means for obtaining viewing data from the user terminal, (c) means for integrating the obtained audience response data and the obtained viewing data in time synchronization, and (d) based on the integrated data, Means for generating audience quality data indicating a viewer's qualitative reaction to the playback content; (e) means for creating viewing metadata for distribution of the generated audience quality data; And a means for transmitting the created viewing metadata to the request source.

  Each of the above processing means can also be realized by a computer program, and this computer program can be provided by being recorded in a recording medium such as a semiconductor memory or via a network.

[3] About the process of the present invention configured as described above In the viewer response information collecting system of the present invention configured as described above, the sensing device obtains the sensing data about the viewer and the recognition process thereof with the sensing device. By collecting either or both of the recognition data of the viewers that are displayed, viewer response data indicating the viewer's response to the playback content is collected. On the other hand, information on the screen operation history is collected at the user terminal. When the viewing data on the playback content including is collected, the collected viewer response data and the collected viewing data are integrated in time synchronization by the user terminal or the viewer response information providing apparatus. Then, the viewer response information providing device generates audience quality data indicating the viewer's qualitative response to the playback content based on the integrated data, and distributes the generated audience quality data for distribution. Create viewing metadata.

  In this way, according to the present invention, it is possible to measure and provide “quality” of viewing “how content (such as a TV program) is viewed” without placing a burden on the viewer. Will be able to.

[4] Specific Configuration of the Present Invention Next, a specific configuration of the present invention will be described.

  Specifically, the present invention is applied to a viewer response information collecting system including a content providing device, a sensing device, a user terminal, and an estimation storage device (corresponding to a viewer response information providing device).

[4-1] Configuration of Content Providing Device The content providing device is constructed by a broadcasting station, an Internet service provider, a content distributor, or the like, and stores content storing means for storing content and content for generating a viewable content list. List generating means, transmitting means for transmitting the generated content list to the user terminal, receiving means for receiving the request information of the content transmitted from the user terminal, and content selection for selecting the content according to the received request information Means, transmitting means for transmitting the selected content to the user terminal, content metadata generating means for generating metadata describing information relating to the content, and transmitting means for transmitting the generated content metadata to the user terminal. .

[4-2] Configuration of Sensing Device The sensing device transmits means for sensing the response of a person viewing content, a recognition means for processing the sensed data, and sensing data and recognition result data. Means.

[4-3] Configuration of User Terminal The user terminal receives a content list transmitted from the content providing apparatus, a content list display unit that outputs the received content list, and a viewing content. Request transmission means for transmitting request information, content reception means for receiving requested content, recording means for buffering received content, reproduction means for playing back received content, and content providing apparatus Receiving means for receiving content metadata, content metadata storing means for storing received content metadata, display means for displaying received content together with received content metadata, screen operation by a user, Entered by user Input means for inputting psychological state information and content evaluation information, screen operation history storage unit for storing screen operation history information operated by the user, and sensing data and recognition data transmitted from the sensing device A data integration means for integrating the input screen operation history information, the user psychological state information, the content evaluation information, and the received sensing data and recognition data with time synchronization, and Transmitting means for transmitting the integrated data to the estimation storage device.

  Here, the user terminal adopts a configuration in which viewing data such as screen operation history information and viewer response data indicating the response of the content viewer transmitted from the sensing device are integrated. You may make it integrate with an apparatus.

[4-4] Configuration of Estimating and Accumulating Device The estimating and accumulating device is a receiving unit that receives integrated data of content transmitted from a user terminal, and a generation that estimates and generates audience quality data of content from the received integrated data. Means, storing audience quality data for storing generated audience quality data, processing means for extracting audience metadata for distribution of audience quality data from the generated audience quality data, and adding new audience metadata Update means and transmission means for transmitting viewing metadata.

[4-5] More Specific Configuration In the viewer reaction information collection system of the present invention configured as described above, information on the viewer's actual viewing time with respect to the playback content and time transition of the viewer's attention level with respect to the playback content Information on viewers' emotions and moods for playback content, information on changes in emotions and moods before and after viewing viewers on playback content, psychological information on playback content input by viewers, viewer input It may include at least one piece of content evaluation information for the playback content.

  In addition, by integrating viewing data such as screen operation history information and viewer response data indicating the response of the content viewer transmitted from the sensing device in a time-synchronized manner, Viewer response to the content of interest can be analyzed in real time.

  Also, even if the content is a TV program recorded on a recording medium, by synchronizing the time based on the content viewing start time, the playback time from the beginning, and the time stamp of the integrated data stored in the estimation storage device , The audience reaction to the content can be analyzed after the fact.

  In addition, the viewer's own emotion and mood are used as the viewer's reaction, and the sensing device senses at least one of the viewer's facial expression, line of sight, blink, body temperature (skin temperature), blood pressure, heart rate, and sweating. Thus, the viewer's own feelings and mood are estimated.

  Also, the viewer's own emotions and moods are used as the viewer's reaction, and the viewer's own emotions and moods such as emotions are estimated according to the input from the viewers.

  Also, a personal information management device comprising means for storing personal information and means for transmitting personal information is prepared, and the user terminal or the estimation storage device accesses the personal information management device, so that the viewer's Personal information may be obtained.

  In the present invention, by employing the above-described technical means, it is possible to automatically determine whether or not the user is actually watching a television program or content. Furthermore, by sensing and recognizing the viewer's reaction at the time of viewing, the viewer's attention level and the mood / emotion estimation for the TV program or content in the normal viewing mode are realized.

  Hereinafter, the structure and operation of the present invention will be described in detail with reference to the accompanying drawings.

  Here, the following description is intended to assist the understanding of the present invention, and does not limit the scope of the claims of the present invention. For example, in the following description, the description is mainly for the case where an infrared light emitting illumination device is used, but the present invention is not limited thereto. Similar effects can be obtained by using visible light and ultraviolet light. Further, the content is not limited to a television broadcast program, and includes digital content and advertisements on the Internet.

  In the embodiment of the present invention, one of the points is to collect and accumulate reaction information of general viewers when watching TV programs and video contents. However, since there is a great psychological resistance to the collection of camera images and microphone sound when generally relaxing at home, the viewer (service user) uses them unless there is any motivation. It is difficult to get permission.

  The first embodiment of the present invention, which will be described later, is greatly characterized in that a general viewer is provided with a profit (hereinafter, “incentive”) as consideration for distributing personal information such as camera video and microphone sound during viewing.

  For example, according to the “Fiber TI Internet” <URL: http://www.pointage.ne.jp/fibertv.html>) 2003.8.25 search, video content such as movies, music, animation, sports, variety, etc. Entertainment content channel “Fibre TB” that delivers on-demand.

  This “fiber tee-by” is a membership-based pay service, and a user who has registered as a member connects a home TV with a set-top box (STB) to view content received via the Internet or a power system CATV. As an incentive for users, points called “views” are earned if advertisements or paid programs are viewed. If you earn this earned point, you can enjoy the privilege that you can watch paid content for free.

  As an example of the incentive, there can be considered a method of making a point that can be exchanged for a free viewing coupon for paid content or a product for a later date, or used for payment of a monthly usage fee for satellite broadcasting or CATV service. It may also be presents by regular lottery or cash back. Such incentives for users are effective, and the business models of “TV program / content audience quality feedback service” and “automatic safety confirmation service for elderly living alone / single householder”, which will be described later, are established.

  FIG. 1 is a diagram for explaining a general configuration example of a system for collecting audience reaction information for content according to the present invention.

  In the figure, 10 is a content providing device, 11 is a content server, 12 is a content metadata DB server, 20 is a user terminal, 21 is a sensing device, 30 is an estimation storage device, 41 is an audience quality DB server, and 42 is a user DB server. 43 are billing settlement servers. A communication network 50 includes a content providing device 10, a content server 11, a content metadata DB server 12, a user terminal 20, a sensing device 21, an estimation storage device 30, an audience quality DB server 41, a user DB server 42, and a billing settlement. Servers 43 are connected to each other. The means for realizing the communication network 50 may be wired or wireless.

  2 and 3 are block diagrams illustrating a typical operation example of the system for collecting viewer response information on content according to the present invention.

  The difference between the two is whether the time synchronization between the content viewing data and the sensing data and recognition data of the viewer response to the content is taken by the user terminal 20 or the estimation storage device 30. That is, FIG. 2 corresponds to the case where the time synchronization is taken on the user terminal 20 side, and FIG. 3 corresponds to the case where the time synchronization is taken on the estimation storage device 30 side.

  4 and 5 are process flow diagrams for explaining the operations of the content providing device 10, the user terminal 20, the sensing device 21, and the estimation storage device 30 according to the present invention. Here, FIG. 4 corresponds to the case where the time synchronization is taken on the user terminal 20 side, and FIG. 5 corresponds to the case where the time synchronization is taken on the estimation storage device 30 side.

  Naturally, the basic processing flow of the present invention is common regardless of the two service application examples described below, “TV program / content audience quality feedback service” and “automatic safety confirmation service for single elderly / single householder”. It has become.

  2 and 3, the content providing apparatus 10 includes a content storage unit 101, a content list generation unit 102, a content list transmission unit 103, a request reception unit 104, a content selection unit 105, a content transmission unit 106, and a content metadata generation unit. 107, a content metadata transmission unit 108.

  The user terminal 20 includes a content list reception unit 2001, a content list display unit 2002, a request transmission unit 2003, a content reception unit 2004, a content recording unit 2005, a content reproduction unit 2006, a screen display unit 2007, a content metadata reception unit 2008, a content A metadata storage unit 2009, a recognition data receiving unit 2010, a data integration unit 2011, an input unit 2012, a screen operation history storage unit 2013, and an integrated data transmission unit 2014 are configured.

  Here, in FIG. 3, the recognition data receiving unit 2010, the data integration unit 2011, and the integrated data transmission unit 2014 are not provided, and a viewing data processing unit 2015 and a viewing data transmission unit 2016 are added instead.

  In the content providing apparatus 10, all usable contents extracted from the content server 11 are temporarily stored in the content storage unit 101. The content list generator 102 identifies video content that can be viewed for each service user, and creates a content list. The content list is transmitted to the user terminal 20 through the content list transmission unit 103.

  In response, in the user terminal 20, the content list receiving unit 2001 receives the content list, and the content list display unit 2002 displays the content list on the terminal screen. When the service user selects a specific content while looking at the content list screen, the location of the requested content, that is, the content URL is extracted from the metadata corresponding to the content (hereinafter referred to as content metadata). The request transmission unit 2003 analyzes the URL of the request content and transmits information necessary for content acquisition, such as a content ID, to the content providing apparatus 10.

  In response to this, in the content providing apparatus 10, the request reception unit 104 receives the content request information from the user terminal 20 and sends it to the content selection unit 105. The content selection unit 105 calls the request content from the content storage unit 101, and the content transmission unit 106 prepares for transmission of the request content.

  At the same time, the content metadata generation unit 107 stores the request content metadata such as title, subtitle, producer, copyright holder, broadcast time / playback time, genre, performer name, (content URL), and the like. It is automatically created from the content metadata DB server 12, and the metadata of the requested content is passed to the content metadata transmission unit 108.

  Subsequently, the request content and the content metadata information thereof are transmitted to the user terminal 20 by the content transmission unit 106 and the content metadata transmission unit 108.

  In response to this, in the user terminal 20, the request content is received by the content receiving unit 2004 and temporarily recorded in the content recording unit 2005 for reproduction. Thereafter, the content requested by the service user is reproduced on the screen by the content reproduction unit 2006.

  On the other hand, the metadata of the request content transmitted together with the request content is received by the content metadata receiving unit 2008, and the metadata information is stored in the content metadata storage unit 2009. Part of the content metadata information such as the title, creator, and time is sent to the screen display unit 2007 and displayed on the terminal screen.

  2 and 3, the sensing device 21 includes a sensing unit 2101, a recognition unit 2102, and a recognition data transmission unit 2103.

  In the present invention, using a noninvasive sensor such as an infrared video camera, a head motion sensor, a microphone, or an RF-ID tag / reader, the facial expression of the person at the time of viewing, the line-of-sight pattern (gaze direction), the pupil diameter of the eye, and blinking Sensing and collecting information such as face orientation, laughter, body temperature, heart rate, and user identification ID.

  As for facial expression recognition of a person using a video camera, a means for quantizing the feature vector of the extracted input image (still image) for each category such as emotions and recognizing the facial expression of each category with high accuracy, for example, Japanese Patent No. 2839855 discloses and discloses a “facial expression recognition device”. As a means for recognizing facial expressions from facial moving images of a large number of unspecified persons, for example, Japanese Patent No. 2962549 describes and discloses a “expression recognition method from facial moving images”.

  In addition, with regard to the gaze direction using an infrared video camera and the measurement of the pupil diameter of one eye, with regard to means for enabling the gaze direction to be measured without burdening the user as much as possible and without wearing a special instrument, For example, Japanese Patent Laid-Open No. 2003-079577 describes and discloses “line-of-sight measurement device and method, line-of-sight measurement program and recording medium recording the program”.

  For detecting laughter using a microphone, a voice processing device that can automatically extract a laughter section with a small amount of computation and a memory size, and means for automatically extracting a laughter section from a moving image For example, Japanese Patent Laid-Open No. 2002-006874 describes and discloses “a sound processing device, a moving image processing device, a sound / moving image processing device, and a recording medium on which a sound / moving image processing program is recorded”.

  Further, the orientation of the face can be easily sensed by attaching a commercially available 3D motion sensor such as MDP-A309 (manufactured by NEC TOKIN) to the head. The body temperature is measured using a commercially available thermometer, and the heart rate is measured by wearing a commercially available heart rate monitor on the chest. Or you may substitute with the pulse rate measured by mounting | wearing a earlobe with a commercially available pulse sensor (for example, Pasorica TP-HB4). The user identification ID may be obtained using a commercially available RF-ID tag and a reader.

  The recognizing unit 2102 recognizes facial expressions, gaze patterns (gaze direction), pupil diameter of eyes, blinks, and laughter that appear on a person's face during viewing using the means described in the above-mentioned patent document. The sensing data and the recognition data acquired in this way are transmitted from the sensing device 21 to the user terminal 20 by the recognition data transmission unit 2103 when the configuration shown in FIG.

  In response, in the user terminal 20, the sensing data and recognition data are received by the recognition data receiving unit 2010 and sent to the data integration unit 2011.

  The data integration unit 2011 synchronizes the sensing data and the recognition data, the service user screen operation history data stored in the screen operation history storage unit 2013, and the content metadata storage unit 2009. The stored content metadata is integrated as integrated data for the requested content (integrated data for viewing and recognition). Note that the screen operation history data described above stores the entire screen operation history input by the service user through the input unit 2012.

  Here, when the input unit 2012 receives psychological state information and content evaluation information input by the user, the data integration unit 2011 also integrates the data.

  The integrated data generated at this time is selected according to the screen operation history data of the service user and what content is selected in what form (for example, the form in which the channel was frequently switched). It is shown how the viewer responded at that time according to the sensing data and the recognition data.

  The integrated data finally synchronized in time is transmitted to the estimation storage device 30 by the integrated data transmission unit 2014 at a certain period when the configuration shown in FIG. 2 is followed.

  In FIG. 2, the estimation storage device 30 includes an integrated data reception unit 301, an audience quality estimation generation unit 302, an audience quality data storage unit 303, an audience quality data transmission unit 304, an audience metadata processing unit 305, and an audience metadata update unit 306. A viewing metadata storage unit 307 and a viewing metadata transmission unit 308.

  Here, in FIG. 3, the integrated data receiving unit 301 is not provided, and the viewing data receiving unit 310, the recognition data receiving unit 311, and the data integrating unit 312 are added in addition to the above 302 to 308.

  In FIG. 2 or 4, the integrated data of the request content is received by the integrated data receiving unit 301 of the estimation storage device 30, and the audience quality estimation generating unit 302 estimates and generates audience quality data described later.

  On the other hand, in FIG. 3 or FIG. 5, the recognition data receiving unit 311 of the estimation storage device 30 receives the viewer response sensing data and recognition data transmitted from the sensing device 21, and receives the viewing data reception of the estimation storage device 30. The unit 310 receives the viewing data obtained by processing the content metadata, the content viewing time, and the user's screen operation history data transmitted from the user terminal 20.

  Then, the data integration unit 312 of the estimation storage apparatus 30 integrates the sensing data and the recognition data of the viewer response to the request content and the viewing data in a time-synchronized manner, and estimates the audience quality. The data is passed to the generation unit 302. Audience quality estimation generation section 302 estimates and generates audience quality data described later.

  Audience quality data in the present invention refers to (1) actual viewing time for viewing content, (2) time transition of attention, (3) emotion for each scene and mood after viewing.

  A specific processing procedure of the audience quality data estimation processing will be described with reference to FIG.

  Regarding (1), it is determined whether the face direction or the line-of-sight position, or both, included in the sensing data and the recognition data are within a predetermined range, and binary time-series data based on the results are respectively determined. Based on this, the actual viewing time for the viewing content is calculated.

  For example, when the time when the line-of-sight position is within the screen range of the display device that is reproducing the content is set as the actual viewing time for the viewing content, the screen range of the display device is determined as the predetermined range and determined to be within that range The actual viewing time for the viewing content can be calculated by counting the time spent.

  The actual viewing time for the viewing content calculated in this way indicates viewing quality, and is one of effective viewing quality data.

Regarding (2), as shown in the process 601 in FIG. 6, the attention level y (t) at time t is determined from the reaction data x _i (t) of n types of viewers at time t and n weighting factors. Using a _i , attention level y (t−1) at time t−1, and one proportional coefficient b,
_{y (t) = a 1 x} 1 (t) + a 2 x 2 (t) + ·· + a n x n (t) + by (t-1)
As described above, it is expressed approximately by a weighted linear sum, and time series data of the value of the degree of attention y (t) is generated. However, the initial value y (0) of the attention level at the initial time is defined as 0.

Here, the viewer's reaction data x _i (t) is set to a numerical value within a predetermined range by using the minimum and maximum values of the sample population collected in advance or the sample mean and sample variance values. Perform normalization.

The viewer response data x _i (t) is selected from the data that is the source of the integrated data, and is set from the sensing data and the recognition data of the viewer response, or the viewer's screen It is set based on the operation history data.

For example, to set the pupil diameter of the eye as x ₁ (t), to set the heart rate as x ₂ (t), sets the frequency of x ₃ (t) as a screen operation (TV remote control). In this case, or spread the pupil, since the heart rate increases is considered physiological phenomena gazing at the content, sets a positive weighting factor a _1, a _2, whereas, frequently operate the TV remote control Therefore, the phenomenon of switching the viewing content is considered not to pay attention to the content. Therefore, by setting a negative weighting factor to a ₃ , the degree of attention y (t) indicating a larger value as the viewer is paying attention Can be defined.

  The temporal transition of the attention level calculated in this way indicates viewing quality, and is one of effective audience quality data.

  With regard to (3), as shown in processing 602 to processing 605 in FIG. 6, first, those having the highest facial expression frequency in a certain time interval are expressed as “joy”, “anger”, “sorrow” as emotions in that time range. ". The time series data of the encoded emotion is used as an emotion for each scene by associating each time interval with the scene.

  Next, the mood for the content is determined using emotion time-series data, or the mood change before and after viewing the content is determined. The mood value is coded into three values of “pleasant”, “unpleasant”, and “indefinite” depending on whether the duration exceeds a predetermined threshold with respect to the emotion value having the maximum duration. For example, if the emotion value with the longest duration is “joy” or “sorrow” and exceeds a predetermined threshold, the mood is “pleasant”, the emotion value with the maximum duration is “anger”, and If the threshold is exceeded, the mood is “uncomfortable”, otherwise “undefined”.

  The emotions and the mood for the content estimated in this way and the change in the mood before and after viewing the content indicate the quality of the viewing and are one of effective audience quality data.

  The audience quality data for the request content thus estimated is stored in the audience quality data storage unit 303 and processed by the viewing metadata processing unit 305 into a format suitable for metadata distribution. When the audience quality data of a service user other than the service user is newly transmitted to the estimation storage device 30 with respect to the request content, it is additionally updated by the viewing metadata update unit 306, and the viewing metadata storage unit 307 is updated. Will be stored.

  FIG. 7 shows an example of the format of viewing metadata processed by the viewing metadata processing unit 305. In this example, the viewing metadata is described on an XML basis, and has high technical compatibility with the unified management in the estimation storage device 30 and the metadata distribution on the network.

  Finally, the audience quality data transmission unit 304 and the viewing metadata transmission unit 308 respond to an audience quality data transmission request and an audience metadata transmission request from a client or a browser, respectively.

  FIG. 8 illustrates an example of a display screen of the actual viewing time measurement function in the audience quality data. FIG. 9 illustrates an example of a display screen example of the attention level estimation function in audience quality data.

  In the screen of FIG. 9, the point is that the time synchronization between the viewing content and the time series data graph of the degree of attention to the viewing content is achieved. That is, by moving the slider of the attention level graph, it is possible for the user to grasp at a glance which scene of the content the attention level is high.

  Although not explained in the above description, when the viewer can be identified, the personal information of the viewer is obtained by accessing a database that manages personal information using the identification result as a key. Then, it may be transmitted in the form of being added to audience quality data or viewing metadata. In this way, it becomes possible to know what kind of viewers have reacted to the content.

[I] First Embodiment of the Present Invention Next, a first embodiment “TV program / content audience quality feedback service” according to the present invention will be described.

  This service is a business model that realizes high-quality content production by feeding back content audience quality information, which has not been possible in the past, to the content producer as a raw opinion on the end user side.

  In the present embodiment, a television program / content producer can obtain data relating to audience quality such as the degree of attention, psychological state, and mood based on the general viewer's reaction to the produced program / content. The audience quality feedback can be used in subsequent programs and content production to differentiate from competitors.

  Specifically, by the time synchronization and integration processing on the user terminal 20 side, the metadata of the content of interest in the estimation storage device 30 and the sensing data relating to various audience reactions of the content transmitted from the sensing device 21・ Recognition data is stored and centrally managed. Therefore, it is possible to view data such as the actual viewing time of the content and the time transition of the attention level at any time using the time stamp as a search key, and further analyze the viewer reaction for each desired scene.

  It should be noted that even for a TV program recorded on a recording medium such as a commercially available HDD recorder or DVD recorder, sensing of various viewer responses of the content of interest transmitted from the sensing device 21 stored in the estimation storage device 30 The time synchronism of the viewing metadata can be maintained by combining the time stamp of the data / recognition data and the content viewing start time and the playback time from the top of the content. Therefore, it is possible to perform post-analysis of various viewer responses to the content offline.

  Furthermore, the advertiser can objectively grasp the effect of the advertisement based on the audience quality data. On the other hand, general viewers can enjoy the benefits of earning points and paying for paid content or viewing paid content for free as consideration for providing facial image and speech (single speech) data.

[II] Second Embodiment According to the Present Invention Next, a second embodiment “automatic safety confirmation service for single elderly / single householder” according to the present invention will be described.

  The service focuses on the fact that most households usually have at least one TV, and provides TV program audience quality information as safety information for singles living alone or family members who are living alone. This is a business model that aims to create a sense of security and a sense of connection.

  “Family Planter” is a system terminal developed in the case of being used by elderly people living alone or single-person householders who live remotely in remote areas, for example, among family members who live remotely in remote areas to be described later. "There is a terminal. Regarding this “Family Planter” terminal, Japanese Unexamined Patent Application Publication No. 2003-141052 (“Intercommunication terminal for human presence / state information, mutual communication method for human presence / state information, mutual communication program, and recording medium recording the program” ) Is described in detail.

  Various sensors are incorporated in the “Family Planter” terminal, and sensor information detected by the sensor is sent to the partner terminal via the network. In the counterpart terminal that has received the sensor information, the motor terminal is rotated according to the received sensor information, and the LED is caused to emit light to display the sensor information. Further, the “Family Planter” terminal is provided with a touch sensor, and when the terminal is touched, sound is generated at the partner terminal in response to the touch, and explicit information can be transmitted.

  In this way, the “Family Planter” terminal acquires the presence / state information of the user mainly by the sensor and sends it to the partner terminal.

  In the second embodiment, the present invention is an image replacing a part of the function of the “Family Planter” terminal. In the following, a detailed description will be given by taking a single employee as an example.

  In the second embodiment, as a basic configuration, by providing television program audience quality information as safety information to a single person living alone or a family head of a single householder, a single person living alone / single person Realize “Automatic safety confirmation service for householders”.

  Furthermore, in the second embodiment, in addition to this basic configuration, the processing as described below can be performed, so that more reliable “automatic safety of single elderly / single householder” Confirmation service "can be realized.

  When the head of the household (user) returns to the home of the single person, the RF-ID reader terminal installed in the room detects the head of the household using the RF-ID tag, and the message of returning home is far away. It is sent to the terminal and displayed on the screen. It is expected that the service user's family will be able to feel the presence of the other party by noticing this display and foster a sense of connection. In addition, by turning on the TV, it is possible to convey the presence of the user to the family living away.

  When the service user desires to connect to the user terminal 20 via the communication line 50, the type of presence / state information that can be used at the terminal on the service user side is acquired from the user terminal 20. At this time, user authentication is performed.

  By this user authentication, the presence / state information that can be used can be changed according to the social familiarity between the user who issued the connection request and the user who is requested to connect. In this way, the privacy problem in the transmission of the presence / state information can be solved to some extent by changing the presence / state information that can be used according to the social familiarity.

  For example, it is assumed that a family living in a remote place can use infrared / ultrasonic sensors, cameras, microphones, and all presence / state information. On the other hand, for users who are concerned only at work, such as nursing care services for elderly people living alone by local governments, there is only the presence / state information obtained from RF-ID tags and infrared sensors. -Status information may be selectively provided.

  Next, an example of an expression format for information on the presence / state information of a person who lives away from the family will be described.

  When the user takes some action or changes in the presence / state of the user, the action and the change are recorded as a time series event and stored in the estimation storage device 30. The presence / state information of a person is expressed by these event groups.

  In addition to text characters, for example, “color” or “sound” can be used as an expression format for notifying the state and safety of the remote party.

  For example, in a simple example using “color”, if there are three types of sensors in the sensing device 21 and they are available, the output signals of these sensors are respectively expressed as three elements of color: red (Red). An example of AD and conversion to green, blue, and blue values is possible.

  On the other hand, a simple example using "sound" is to automatically call a pre-registered phone number (home) and make a ringing tone only three times, or automatically download a WAV file registered in advance on a PC screen of a distant family Or play it.

  In this way, according to the second embodiment “automatic safety confirmation service for the elderly living alone / single householder” according to the present invention, the TV program (content) producer can view the produced program (content) in general. Data related to audience quality such as attention, psychological state, and mood based on the person's reaction. Then, the audience quality feedback can be used for subsequent program (content) production to differentiate from competitors.

  In addition to this, the elderly living alone or a family member / relative of a single household can check the safety of the elderly living alone or a family who is currently on their own without being aware of the person to be confirmed based on the audience quality data. A casual watch over the network can be realized. On the other hand, the elderly living alone / single-person householder can enjoy the benefits of collecting points or paying for viewing paid content free of charge for providing facial images and spoken voice (single speech) data.

  As described above, the viewer response information collection system for content according to the present invention is a multimodal system such as face orientation by 3D motion sensor, human eye gaze pattern by infrared video camera, pupil diameter, facial expression, laughter. By integrating the information, the workload such as entry by the viewer himself / herself and answering on the web is reduced, and automatic audience quality data in the normal viewing mode is realized. At the same time, it can be expected that an individual viewer can be identified quickly and easily by reading the RF-ID tag.

  In addition, by the time synchronization and integration processing in the user terminal 20 and the estimation storage device 30, various viewing of the metadata of the content of interest and the content transmitted from the sensing device 21 in the estimation storage device 30. Sensing data / recognition data related to person reactions are accumulated and managed as viewing metadata information. Therefore, it is possible to analyze the viewer reaction for each desired scene at any time using the time stamp as a search key.

  Furthermore, even for a TV program recorded on a recording medium such as an HDD or a DVD, sensing of various viewer responses of the content of interest transmitted from the sensing device 21 by time synchronization and integration processing on the estimation storage device 30 side. The data / recognition data is given a time stamp. Therefore, by combining the time stamp with the content viewing start time and the playback time from the top of the content, it becomes possible to perform post-analysis of various viewer responses to the content of interest.

  If the present invention is applied to an audience quality feedback service, a television program / content producer can produce programs / contents utilizing audience quality information, which leads to an increase in profits of the company.

  If the present invention is applied to a remote resident's automatic safety confirmation service, through the acquisition of audience quality information, the safety can be confirmed in an infinitely natural form or watched without the resident himself being conscious. Is possible. In other words, through the network, your terminal and partner in a form that blends into the living environment without making the other party aware of the information such as the presence of people, living well, or being healthy. There is an effect that it is possible to confirm safety by constantly exchanging with the terminal and to foster a sense of connection between users.

It is a figure explaining the general structural example of the collection system of the audience reaction information with respect to the content of this invention. It is a block diagram explaining the typical operation example of the collection system of the audience reaction information with respect to the content of this invention. It is a block diagram explaining the typical operation example of the collection system of the audience reaction information with respect to the content of this invention. It is a processing flowchart explaining operation | movement of the content provision apparatus in this invention, a user terminal, a sensing apparatus, and an estimation storage apparatus. It is a processing flowchart explaining operation | movement of the content provision apparatus in this invention, a user terminal, a sensing apparatus, and an estimation storage apparatus. It is a block diagram explaining operation | movement of the audience quality estimation production | generation part of this invention. It is the figure which showed an example of viewing-and-listening metadata information. It is the figure which showed an example of the display screen of a real viewing-and-listening time measurement function. It is the figure which showed an example of the display screen of an attention degree estimation function.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Content provision apparatus 11 Content server 12 Content metadata DB server 20 User terminal 21 Sensing apparatus 30 Estimating storage apparatus 41 Audience quality DB server 42 User DB server 43 Charge settlement server

Claims (15)

A viewer response information collecting system that collects viewer response information for content to be played back,
Viewer response data indicating the viewer's response to the playback content by collecting one or both of sensing data about the viewer and recognition data about the viewer obtained by performing recognition processing on the viewer. A sensing device that collects
While reproducing the content, collecting viewing data regarding the reproduced content including information on the screen operation history, and further obtaining the viewer response data from the sensing device, and collecting the collected viewing data and the obtained viewer A user terminal that integrates reaction data in time synchronization;
Obtaining the integrated data from the user terminal, generating audience quality data indicating a qualitative reaction of the viewer to the playback content based on the obtained integrated data, and generating the audience quality data A viewer response information providing device for creating viewing metadata for distribution of
A characteristic audience response information collection system. A viewer response information collecting system that collects viewer response information for content to be played back,
Viewer response data indicating the viewer's response to the playback content by collecting one or both of sensing data about the viewer and recognition data about the viewer obtained by performing recognition processing on the viewer. A sensing device that collects
A user terminal that reproduces the content and collects viewing data about the reproduced content including information on the screen operation history;
The viewing data is obtained from the user terminal, the viewer response data is obtained from the sensing device, and the obtained viewing data and the obtained viewer response data are integrated in time synchronization. Viewer response information is provided to generate audience quality data that indicates the viewer's qualitative reaction to the playback content based on the recorded data and to create viewing metadata for distribution of the generated audience quality data Comprising a device,
A characteristic audience response information collection system. A viewer response information collection method for collecting viewer response information for content to be played back,
Viewer response data indicating the viewer's response to the playback content by collecting one or both of sensing data about the viewer and recognition data about the viewer obtained by performing recognition processing on the viewer. The process of collecting
A process of collecting viewing data about playback content including information on screen operation history,
Integrating the viewer response data and the viewing data in time synchronization;
Generating audience quality data indicating a qualitative reaction of the viewer to the playback content based on the integrated data, and creating viewing metadata for distribution of the generated audience quality data. That
A feature of audience response information collection method. In the viewer reaction information collection method according to claim 3,
In the process of creating, the viewer's actual viewing time information for the playback content, the viewer's attention level transition information for the playback content, the viewer's emotion / mood information for the playback content, and the viewer's attention to the playback content Audience quality data including at least one of emotion / mood change information before and after viewing, information on psychological state with respect to playback content input by the viewer, and content evaluation information with respect to playback content input by the viewer. Generating and creating viewing metadata for distribution of the generated audience quality data,
A feature of audience response information collection method. In the viewer reaction information collection method of Claim 3 or 4,
In the integration process, in order to realize real-time creation of the viewing metadata, integrating the viewer response data and the viewing data in a time-synchronized manner in the form of sequentially combining both data,
A feature of audience response information collection method. In the viewer reaction information collection method of Claim 3 or 4,
In the integrating step, a time stamp assigned when the viewer response data is collected and a time stamp assigned when the viewing data is collected in order to realize post-creation of the viewing metadata. And integrating the viewer response data and the viewing data in time synchronization with each other,
A feature of audience response information collection method. In the viewer reaction information collection method according to any one of claims 3 to 6,
In the process of collecting the viewer response data, in order to collect information indicating the viewer's emotion and mood, the viewer response data includes the viewer's facial expression, line of sight, blink, body temperature, blood pressure, heart rate, sweating. Collecting at least one of them,
A feature of audience response information collection method. In the viewer reaction information collection method according to any one of claims 3 to 6,
In the process of collecting the viewing data, in order to collect information indicating the viewer's emotion / mood, information indicating the viewer's emotion / mood is input from the input device of the user terminal as one of the viewing data. To collect,
A feature of audience response information collection method. In the viewer reaction information collection method according to any one of claims 3 to 8,
Including a process of acquiring personal information of a viewer who views the playback content that is the creation source of the viewing metadata from an apparatus that manages personal information.
A feature of audience response information collection method. A user terminal used in a viewer response information collection system that collects viewer response information for content to be played back,
Means for reproducing the content of the viewing request transmitted from the content providing device;
Means for obtaining metadata of playback content transmitted from the content providing device;
Means for collecting screen operation history information by obtaining screen operation information by a viewer and storing it in a storage means;
Viewer response data indicating the viewer's response to the playback content by collecting one or both of sensing data about the viewer and recognition data about the viewer obtained by performing recognition processing on the viewer. Means for obtaining the viewer response data from a sensing device that collects
Means for integrating the screen operation history information and the viewer response data in time synchronization, and integrating the metadata to it;
Means for transmitting the integrated data to a viewer response information providing device that provides information on viewer response to content.
A user terminal used in a characteristic audience reaction information collecting system. A viewer response information providing apparatus used in a viewer response information collecting system for collecting information of a viewer response to reproduced content,
Integrated data generated by integrating screen operation history information and viewer response data indicating the viewer's response in time synchronization from the user terminal that plays back the content, and by integrating the metadata of the playback content into it A means of obtaining
Means for generating audience quality data indicating a qualitative reaction of the viewer to the playback content based on the integrated data;
Means for creating viewing metadata for distribution of the audience quality data;
Means for transmitting the viewing metadata to a request source,
A viewer response information providing apparatus used in a characteristic viewer response information collecting system. A user terminal used in a viewer response information collection system that collects viewer response information for content to be played back,
Means for reproducing the content of the viewing request transmitted from the content providing device;
Means for obtaining metadata of playback content transmitted from the content providing device;
Means for collecting screen operation history information by obtaining screen operation information by a viewer and storing it in a storage means;
Means for creating viewing data by processing the information of the screen operation history and the metadata;
Means for transmitting the viewing data to a viewer response information providing device that provides viewer response information to the content,
A user terminal used in a characteristic audience reaction information collecting system. A viewer response information providing apparatus used in a viewer response information collection system for collecting information of a viewer response to content to be played back,
Viewer response data indicating the viewer's response to the playback content by collecting one or both of sensing data about the viewer and recognition data about the viewer obtained by performing recognition processing on the viewer. Means for obtaining the viewer response data from a sensing device that collects
Means for obtaining viewing data generated by processing screen operation history information and reproduction content metadata from a user terminal that reproduces content;
Means for synchronizing the viewer response data and the viewing data in time synchronization;
Means for generating audience quality data indicating a qualitative reaction of the viewer to the playback content based on the integrated data;
Means for creating viewing metadata for distribution of the audience quality data;
Means for transmitting the viewing metadata to a request source,
A viewer response information providing apparatus used in a characteristic viewer response information collecting system. A program for creating viewer reaction information for causing a computer to execute processing used to realize the user terminal according to claim 10. A program for creating viewer response information for causing a computer to execute processing used to realize the viewer response information providing apparatus according to claim 11.

Images