JP2017188715A - Video display system and video display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a visual observation object of a spectator.SOLUTION: A visual observation object detection unit 11 analyzes face picture information on a spectator 3 imaged by a second imaging device 40 to detect visual observation direction 3e of the spectator 3. A position detection device 50, on the basis of position information on a seat inside a spectator seat area 2, detects viewpoint position information on the spectator 3. The visual observation object detection unit 11, on the basis of viewpoint position information on the spectator 3, visual observation direction 3e of the spectator 3, and position information on a playing field 1a and playing field 1b, detects a visual observation object area to be visually observed by the spectator 3. A display control unit 12, when the visual observation object area is overlapped with the playing field 1a, outputs a video signal of a first imaging device 31 for imaging the playing field 1a to a display unit 13. When the visual observation object area is overlapped with the playing field 1b, the display control unit outputs a video signal of a first imaging device 32 for imaging the playing field 1b to the display unit 13.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video display system and a video display method.

  2. Description of the Related Art In recent years, video display devices that display a state of competition on a large screen image are sometimes installed in places where spectators gather such as sports competition venues. The display target to be displayed on such a large screen image is appropriately selected according to, for example, a cameraman who performs shooting or an instruction registered in advance in the image display device. Some of such video display devices divide a display screen and display images taken by a plurality of imaging devices.

However, in the conventional video display device, since a cameraman or the like has selected a display target of a large screen video, there is a case where a target that many spectators want to see is not displayed on the large screen.
In view of this, a technique has been proposed in which an object focused by the audience is detected and the detected object is displayed on a large screen image. For example, there is a video display system in which a plurality of display objects are displayed on a split display screen, and an imaging device for capturing a spectator who is watching the split display screen is provided. The imaging device detects the viewing direction of the audience based on the captured image obtained by capturing the audience, and displays the screen on which the audience's interest is gathered among the divided display screens on the main screen. In addition, a plurality of spectators who watch sports matches are equipped with gaze input devices, and based on the result of statistical processing of the detected gaze of the spectators, an image in which many spectators are interested is cut out There is a video cutout control device. The line-of-sight input device detects a line-of-sight position viewed by the audience. Further, there is a technique for detecting the size of the pupil of the subject and detecting the psychological state of the subject when detecting the line of sight.

JP 2011-146048 A JP 2003-125286 A Japanese Patent Laid-Open No. 2008-3816

  Unlike when the subject is viewing the screen in the vicinity of the visual target, such as a video display device, in a gymnasium or stadium, which is a sports competition venue, the audience is in a wide range away from the target area to be viewed. . For this reason, even if it is the same visual object, the visual direction detected by the position of an audience will differ. Therefore, it is not possible to specify the object that the audience is looking at simply by detecting the viewing direction. If the spectator wears a gaze input device, the gaze position of the spectator can be detected, but it is not realistic to wear the gaze input device and watch the game because it is expensive.

As described above, it is not easy to detect the visual target of the spectator in a place where there is a spectator who views the target region from multiple directions as in a sports venue.
In one aspect, the present invention provides a video display system and a video display method capable of detecting a visual target of a spectator and displaying a video.

  In one aspect, a video display system including a plurality of first imaging devices, a second imaging device, a position detection device, and a video display device is provided. A 1st imaging device is arrange | positioned at each of several target area | region, and image | photographs the arrange | positioned target area | region. The second imaging device captures a face image of a spectator who views a plurality of target areas. The position detection device detects the viewpoint position of the audience. The video display device includes a visual object detection unit and a display control unit. The visual target detection unit detects the viewing direction of the spectator based on the spectator's face image, and the sighting direction of the spectator, the viewpoint position information of the spectator detected by the position detection device, and the positional information of the plurality of target regions. Based on the plurality of target areas, a visual target area that the audience is viewing is detected. The display control unit displays, on the display unit, an image captured by the first imaging device corresponding to the viewing target region among the plurality of first imaging devices.

  In one aspect, a video display method executed by a computer is provided.

  According to one aspect, in the place where the audience views the target area from multiple directions, the visual target area viewed by the audience can be detected and an image obtained by photographing the visible area can be displayed.

It is a figure which shows the structure of the video display system of 1st Embodiment. It is a figure which shows the structure of the video display system of 2nd Embodiment. It is a figure which shows the hardware structural example of an image analysis apparatus. It is a block diagram which shows the function structure of the video display system of 2nd Embodiment. It is a figure which shows an example of game hall information. It is a figure which shows an example of audience information. It is a figure which shows the outline of an audience visual object detection process. It is a flowchart which shows the procedure of an audience visual object detection process. It is a flowchart which shows the procedure of a visual object area | region detection process. It is a flowchart which shows the procedure of a precursor operation | movement detection process. It is a flowchart which shows the procedure of an attention degree calculation process. It is a figure which shows the structure of the video display system of 3rd Embodiment. It is a block diagram which shows the function structure of the video display system of 3rd Embodiment. It is a figure which shows an example of the division | segmentation display screen of the video display system of 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration of a video display system according to the first embodiment. The video display system according to the first embodiment is installed in a venue where there are a plurality of target areas to be viewed by a spectator. For example, a sports venue where a plurality of game venues are installed.

  In the example of FIG. 1, the video display system is installed in a venue where a spectator seat area 2 is provided around the game field 1. The game field 1 includes two target areas, a game field 1a and a game field 1b. A spectator 3 is seated in a spectator seat area 2 around the game field 1. The spectator 3 views the game being played on the game field 1a or the game field 1b. The target area viewed by the spectator 3 changes according to the interest of the spectator 3. Hereinafter, the target area that the spectator 3 is viewing is referred to as a visual target area.

The video display system includes a video display device 10, first imaging devices 31 and 32, a second imaging device 40, and a position detection device 50.
The first imaging devices 31 and 32 are respectively arranged in the target area, take pictures of the arranged target areas, and output the taken video signals to the video display apparatus 10.

The second imaging device 40 images the spectator 3 seated in the spectator seat area 2. The second imaging device 40 captures the audience 3 and outputs face image information of the captured audience 3 to the video display device 10.
The position detection device 50 detects the viewpoint position of the spectator 3 and outputs the detected viewpoint position information to the video display device 10. For example, the position detection device 50 images the spectator seat area 2 from above and analyzes the image to detect whether a person is seated. Alternatively, it may be a sensor that is provided for each spectator seat and detects whether a person is seated. The position detection device 50 generates the viewpoint position information of the seated audience based on the position coordinates defined in advance for each audience seat.

  The video display device 10 includes a visual target detection unit 11, a display control unit 12, and a display unit 13, acquires video signals from the first imaging devices 31 and 32, and displays the game field 1 a or the game field 1 a on the monitor screen 13 a of the display unit 13. The video of the game field 1b is displayed. The visual target detection unit 11 and the display control unit 12 are, for example, a processor such as a CPU (Central Processing Unit). However, the visual target detection unit 11 and the display control unit 12 may include an electronic circuit for a specific application such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Note that the video display device 10 may be composed of a plurality of devices.

  The visual object detection unit 11 analyzes the face image information of the audience 3 acquired from the second imaging device 40 and detects the viewing direction of the audience 3. The viewing direction indicates which direction the audience 3 is facing with respect to a reference direction, for example, the front direction of the audience 3. The visual object detection unit 11 allows the spectator 3 to visually check based on the detected viewing direction of the spectator 3, the viewpoint position information of the spectator 3 acquired from the position detection device 50, and the positional information of the game field 1a and the game field 1b. Detecting the visual target area. The same processing is performed for each of the plurality of spectators 3 to detect the visual target area.

  The display control unit 12 determines a target region for displaying an image on the display unit 13 based on the visual target region of the spectator 3 detected by the visual target detection unit 11. When there are a plurality of spectators 3, the display control unit 12 performs a statistical process to determine a target area to be displayed. For example, the number of spectators visually observed for each target area is totaled, and the target area with the largest number of spectators is selected. The display control unit 12 outputs video signals captured by the first imaging devices 31 and 32 corresponding to the selected target area to the display unit 13.

The display unit 13 has a monitor screen 13a and displays an image of the game field 1a or the game field 1b based on the video signal acquired from the display control unit 12.
The operation of the video display system having such a configuration will be described using the example shown in FIG.

  The game field 1 has a game field 1a and a game field 1b. A first image pickup device 31 is arranged in the game field 1a, and a first image pickup device 32 is arranged in the game field 1b. The spectator 3 is seated in the spectator seat area 2 and is watching a competition that takes place in the game field 1a or the game field 1b. In the audience seat area 2 or in the vicinity of the audience seat area 2, a second imaging device 40 that photographs the audience 3 and a position detection device 50 that detects the position of the audience 3 are arranged.

  The first imaging device 31 outputs a video signal obtained by shooting the game field 1 a to the video display device 10. The 1st imaging device 32 outputs the video signal which image | photographed the game field 1b to the video display apparatus 10. FIG. The second imaging device 40 outputs a face image obtained by photographing the audience 3 to the video display device 10. The position detection device 50 detects the position of the seat where the spectator 3 is seated, and outputs the detected viewpoint position information of the spectator to the video display device 10.

  In the video display device 10, the visual target detection unit 11 detects the visual target region of the spectator 3, and the display control unit 12 executes a process of determining the target region to be displayed on the monitor screen 13a at a predetermined cycle.

  The visual object detection unit 11 detects the visual direction 3e of the audience 3 by detecting the direction of the face of the audience 3 and the direction of the eyeball based on the face image information of the audience 3. The viewing direction is represented by an angle from the reference, for example, based on the state when the spectator 3 is seated and looks at the front. Note that depending on the positional relationship between the audience 3 and the second imaging device 40, the face image of the audience 3 cannot be captured from the front. The visual object detection unit 11 corrects the direction of the face and the direction of the eyeball based on the positional relationship between them to obtain the visual direction 3e. The visual target detection unit 11 detects the visual target area of the spectator 3 based on the detected visual direction 3e of the spectator 3, the viewpoint position information of the spectator 3, and the positional information of the game field 1a and the game field 1b. In the example of FIG. 1, the game field ahead of the viewing direction 3e of the spectator 3 is detected. Here, the visual target detection unit 11 determines that the visual target area of the spectator 3 is the game field 1a. The visual target detection unit 11 outputs the detected visual target region of the spectator 3 to the display control unit 12.

  The display control unit 12 selects a video signal of the first imaging device 31 that captures the visual target area based on the visual target area of the spectator 3 and outputs the video signal to the display unit 13. When the viewing target areas of a plurality of spectators are detected, statistical processing is performed to determine the viewing target area to be output to the display unit 13. For example, a viewing target region that is viewed by the largest number of spectators is selected. The display unit 13 displays a video based on the input video signal on the monitor screen 13a. In the example of FIG. 1, the display unit 13 displays an image of the game field A (1a) on the monitor screen 13a.

  Thereby, when the spectator 3 is viewing the game field 1a, the video of the game field 1a is displayed on the monitor screen 13a. Further, when the spectator 3 is viewing the game field 1b, the display on the monitor screen 13a is switched to the video image of the game field 1b. When there are a plurality of spectators 3, statistical processing is performed on the viewing target areas detected for each spectator. For example, the images of the first imaging devices 31 and 32 that capture the target areas viewed by the largest number of spectators 3. The signal is selected and displayed on the monitor screen 13a.

  In the video display system according to the first embodiment, the position and viewing direction of the spectator are detected, and the visual target area that the spectator is looking at is detected. Since the visual target area is detected for each spectator and the target area to be displayed on the monitor screen 13a is determined based on the detected visual target area, a video of the target area in which many spectators are interested is displayed on the monitor screen 13a. can do.

[Second Embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram illustrating a configuration of a video display system according to the second embodiment. The video display system according to the second embodiment includes a first game field 101, a second game field 102, a third game field 103, and a fourth game field 104 that can be played simultaneously. It is applied to the competition venue where the spectator seat 120 is formed. For example, there are competition venues such as judo, kendo, and gymnastics. The spectator seat 120 is arranged at a predetermined position. The spectator 130 sits on the spectator seat 120 and watches a game of interest among the games played in the first game field 101, the second game field 102, the third game field 103, and the fourth game field 104. .

  The video display system includes game cameras 211, 212, 213, 214, passenger seat cameras 221, 222, ceiling cameras 231, 232, an image analysis device 300, an output video creation device 400, and a monitor device 500. Each device is connected via a network. The image analysis device 300 and the output video creation device 400 are, for example, computers.

  The game camera 211 is arranged in the first game field 101, the game camera 212 is arranged in the second game field 102, the game camera 213 is arranged in the third game field 103, and the game camera 214 is arranged in the fourth game field 104, respectively. The game cameras 211, 212, 213, and 214 capture the arranged game field and transmit the video signal to the output video creation device 400. The game cameras 211, 212, 213, and 214 are examples of the first imaging devices 31 and 32 of the first embodiment.

  The passenger seat cameras 221 and 222 capture the face image of the spectator 130 seated on the spectator seat 120 and transmit the face image data to the image analysis apparatus 300. For example, the audience cameras 221 and 222 may be arranged so that all the spectators 130 can be photographed, or some spectators 130 may be arranged as a sample so as to be photographed. The audience cameras 221 and 222 photograph the audience 130 in a predetermined range at a predetermined cycle. The passenger seat cameras 221 and 222 are an example of the second imaging device 40 of the first embodiment. The arrangement of the two passenger seat cameras 221 and 222 is an example, and the number of the cameras to be arranged is determined as appropriate.

  The ceiling cameras 231 and 232 photograph the spectator seat 120 from above, and transmit the occupant seat image data to the image analysis device 300. A predetermined number of ceiling cameras are arranged at predetermined locations so that the entire range of audience seats 120 can be photographed. When some of the spectator seats 120 are photographed as samples, the ceiling cameras 231 and 232 are arranged so that the photographing area is the same area as the area photographed by the occupant seat cameras 221 and 222. The ceiling cameras 231 and 232 photograph the audience seat 120 in a predetermined range at a predetermined cycle. The ceiling cameras 231 and 232 are an example of the position detection device 50 according to the first embodiment. The arrangement of the two ceiling cameras 231 and 232 is an example, and the number of the ceiling cameras 231 and 232 is appropriately determined.

  The image analysis device 300 analyzes the face image data of the audience 130 acquired from the audience cameras 221 and 222 and the audience image data acquired from the ceiling cameras 231 and 232, and detects the viewing direction and viewpoint position of the audience 130. The image analysis apparatus 300 compares the viewing direction and viewpoint position with the position coordinates of each game field, and detects the game field that the spectator 130 is viewing. In the second embodiment, the viewing target area of the spectator 130 is calculated as a positional relationship on the two-dimensional plane indicated by the seat image data of the ceiling cameras 231 and 232. In the following, the positional relationship will be discussed on this two-dimensional plane, and the depth direction in FIG. 2 will not be mentioned. Further, in the following, the position coordinates of each of the game fields 101, 102, 103, 104, the spectator seat 120, the spectator cameras 221, 222, and the ceiling cameras 231, 232 are based on an arbitrary position on the two-dimensional plane as a reference point. Each position is indicated. The image analysis apparatus 300 analyzes the face image data of the plurality of spectators 130 in the spectator seat 120 and performs statistical processing to detect a game in which the most spectators are interested. Further, the image analysis apparatus 300 may detect the order of the game in which the audience is interested. The image analysis apparatus 300 transmits the game field that the most spectators 130 are interested in to the output video creation apparatus 400.

  The output video creation device 400 inputs video signals of each game field taken by the game cameras 211, 212, 213, 214. The output video creation device 400 acquires information about the game field that the spectator 130 is interested in from the image analysis device 300, and outputs a video signal of the corresponding game field to the monitor device 500.

The monitor device 500 displays a video based on the output video signal input from the output video creation device 400 on the monitor screen.
FIG. 3 is a diagram illustrating a hardware configuration example of the image analysis apparatus. The entire image analysis apparatus 300 is controlled by the CPU 301. A RAM (Random Access Memory) 302 and a plurality of peripheral devices are connected to the CPU 301 via a bus 308.

  The RAM 302 is used as a main storage device of the image analysis device 300. The RAM 302 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the CPU 301. The RAM 302 stores various data necessary for processing by the CPU 301.

  Peripheral devices connected to the bus 308 include an HDD (Hard Disk Drive) 303, a graphic processing device 304, an input interface 305, an optical drive device 306, and a communication interface 307.

  The HDD 303 magnetically writes data to and reads data from a built-in disk. The HDD 303 is used as a secondary storage device of the image analysis device 300. The HDD 303 stores an OS program, application programs, and various data. Note that a semiconductor storage device such as a flash memory can also be used as the secondary storage device.

  A monitor 311 is connected to the graphic processing device 304. The graphic processing device 304 displays an image on the screen of the monitor 311 in accordance with a command from the CPU 301. Examples of the monitor 311 include a display device using a CRT (Cathode Ray Tube) and a liquid crystal display device.

  A keyboard 312 and a mouse 313 are connected to the input interface 305. The input interface 305 transmits a signal transmitted from the keyboard 312 or the mouse 313 to the CPU 301. Note that the mouse 313 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

  The optical drive device 306 reads data recorded on the optical disk 314 using laser light or the like. The optical disk 314 is a portable recording medium on which data is recorded so that it can be read by reflection of light. The optical disk 314 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), and the like.

  The communication interface 307 is connected to the network 600. The communication interface 307 transmits / receives data to / from other computers or communication devices via the network 600.

  With the hardware configuration as described above, the processing functions of the present embodiment can be realized. 3 shows the hardware configuration of the image analysis apparatus 300, the output video creation apparatus 400 also has the same hardware configuration.

FIG. 4 is a block diagram illustrating a functional configuration of the video display system according to the second embodiment.
The game cameras 211,..., 214 shoot a game played on the arranged game field and output the photographed video signal to the output video creation device 400. The video signal is used for display on the monitor device 500.

  The passenger seat cameras 221 and 222 capture the face of the spectator 130 at a predetermined cycle, and output the face image data to the precursor motion detection unit 310 and the spectator visual direction detection unit 320. The face image data may be stored in the storage unit or storage unit 350 in the passenger seat cameras 221 and 222 and read by the precursor motion detection unit 310 and the spectator visual direction detection unit 320 when necessary.

  The ceiling cameras 231 and 232 photograph the spectator seat 120 at a predetermined cycle, and output the photographed spectator seat image data to the spectator seating position detection unit 330. The ceiling cameras 231 and 232 generate audience image data including all the audience seats 120. The audience image data may be stored in the storage unit or storage unit 350 in the ceiling cameras 231 and 232, and the audience seating position detection unit 330 may read it out when necessary.

  The image analysis apparatus 300 includes a precursor motion detection unit 310, a spectator visual direction detection unit 320, a spectator seating position detection unit 330, a spectator visual object detection unit 340, a storage unit 350, and an attention level calculation unit 360. Have. The precursor motion detection unit 310, the audience viewing direction detection unit 320, the audience seating position detection unit 330, the audience viewing object detection unit 340, and the attention level calculation unit 360 can be realized as modules of a program executed by the CPU 301. The storage unit 350 can be realized as a storage area secured in the RAM 302 or the HDD 303.

  The precursor motion detection unit 310, the spectator visual direction detection unit 320, and the spectator seating position detection unit 330 are activated at predetermined intervals, and the face image data and the spectator image data captured by the spectator cameras 221 and 222 and the ceiling cameras 231 and 232 are used. To analyze.

  The sign motion detection unit 310 analyzes the face image data of the spectator 130 and detects a motion that can be a sign of appearance of a visual target region where the visual targets of many spectators 130 gather. Hereinafter, the operation detected at this time is referred to as a precursor operation. For example, when an event that attracts the interest of many spectators 130 occurs, such as during a sporting game match, the spectators 130 rarely watch the event all at once. There are many cases where some spectators 130 see the event and it spreads around. In view of this, the fact that a part of the spectator has focused on a certain game field is detected as a precursor action and is referred to when selecting a video signal to be displayed on the monitor device 500. For example, when it is found that some spectators are interested in a game field, the display screen of the monitor device 500 is switched to this game field, thereby quickly detecting a visual object that the spectator is interested in. And can be reflected in the display.

  Here, as an operation when an event that attracts interest from the spectator 130 is detected, for example, the viewing direction may be turned there, and an operation such as opening the eyes, changing the pupil, or raising a cheer may be involved. In the second embodiment, a precursor motion is detected based on changes in the pupil. It is known that when an event that attracts interest is detected, the psychological state of the subject can be estimated from the facial expression of the subject, particularly the eye state. For example, it is known that a person's pupil is large when the person is relaxed and small when he is nervous or surprised. It is also said that the pupil becomes larger when looking at something of interest. Therefore, by detecting a change in the size of the pupil of the audience 130, it is possible to detect that an event that attracts the audience's interest has occurred. The sign motion detection unit 310 detects the size of the pupil of the spectator 130 when analyzing the face image data, and stores it in the storage unit 350. The sign motion detection unit 310 compares the pupil size of the spectator 130 detected at the time of the next facial image data analysis with the pupil size stored in the storage unit 350 to detect a change. When the sign motion detection unit 310 detects a change in the size of the pupil, the sign motion detection unit 310 notifies the audience visual target detection unit 340 that the sign motion has been detected. For example, the warning motion detection unit 310 stores the detection information of the warning motion in the storage unit 350 that can be read from the audience visual object detection unit 340.

  The audience viewing direction detection unit 320 sequentially analyzes the face image data of the audience 130 taken by the audience seat cameras 221 and 222. The audience viewing direction detection unit 320 detects the direction of the face and the direction of the eyeball from the face image data of the audience 130, and detects the viewing direction. For example, the viewing direction is expressed as an angle formed by the viewing direction and the front, with the state where the audience 130 of the spectator seat 120 is looking at the front as a reference (0 °). The audience viewing direction detection unit 320 analyzes the face image data for each audience 130 to be detected, and detects the viewing direction.

  The audience seating position detection unit 330 determines whether or not the audience 130 is seated based on the audience image data of the audience seat 120 taken by the ceiling cameras 231 and 232. The spectator seating position detection unit 330 is seated when, for example, a color different from the color of the seat surface or the color of the hair is detected at a point corresponding to the spectator seat position (position coordinates) of the spectator seat 120 defined in advance. It is determined that The audience seating position detection unit 330 sets the position coordinates where the audience 130 is seated as the audience seating position of the audience 130. The audience seating position can be regarded as the viewpoint position coordinates of the audience 130 on a two-dimensional plane.

  The spectator visual target detection unit 340 detects the visual target region of the spectator based on the viewpoint position coordinates and the visual direction of the spectator 130 detected by the spectator visual direction detection unit 320 and the spectator seating position detection unit 330. The spectator visual object detection unit 340 collates the viewpoint position coordinates and viewing direction of the spectator 130 with the position coordinates of each game field, detects a game field where the visual direction of the spectator 130 overlaps with the area, and detects the detected game field. It is set as a viewing target area of the audience 130.

The storage unit 350 stores game venue information 351 and spectator information 352.
The game hall information 351 is information regarding the game hall, and is referred to when the spectator visual target detection unit 340 detects the visual target area. Details will be described later with reference to FIG. The audience information 352 is information related to the audience seat 120 and the audience 130 and is referred to by the precursor motion detection unit 310, the audience visual direction detection unit 320, and the audience seating position detection unit 330. Details will be described later with reference to FIG.

  The attention level calculation unit 360 statistically processes the visual target area of the spectator 130 detected by the spectator visual target detection unit 340, and calculates the attention level. The attention level calculation unit 360 calculates, for example, the number of spectators 130 viewing the game field for each game field, and sets it as the attention level. Further, the ratio of the calculated audience number to the total audience number may be used as the degree of attention. When calculating the number of spectators, the degree-of-attention calculation unit 360 performs weighting to increase the degree of attention of the visual target area viewed by the spectator 130 in which the precursor motion is detected. Thereby, the attention degree of the game field which the spectator 130 in which the warning motion is detected can be raised. The attention level calculation unit 360 notifies the output video creation device 400 of information on the game field with the highest level of attention.

  The output video creation device 400 includes a display video creation unit 410. The display video creation unit 410 inputs video signals of the first game field 101, the second game field 102, the third game field 103, and the fourth game field 104 taken by the game cameras 211,. Further, the display video creation unit 410 acquires information on the game field with the highest degree of attention from the degree-of-attention calculation unit 360. The display video creation unit 410 selects a video signal corresponding to the acquired game field with the highest degree of attention and outputs it to the video output unit 510.

  The monitor device 500 includes a video output unit 510 and a display unit 520. The video output unit 510 converts the video signal acquired from the display video creation unit 410 into a display signal for display, and outputs the display signal to the display unit 520. Display unit 520 displays the video signal of the selected game field based on the display signal.

FIG. 5 is a diagram illustrating an example of game venue information.
The game hall information 351 includes information items of “game field”, “camera”, and “position” as information on the game field.

  In the “game field”, a game field identification number for identifying the first game field 101, the second game field 102, the third game field 103, and the fourth game field 104 is registered. In the “camera”, a camera identification number for identifying the game cameras 211, 212, 213, 214 arranged on the game field is registered in association with the game field identification number. In “position”, the position coordinates of the game field are registered in association with the game field identification number. The position coordinates of each game field indicate the four corner positions of the game field.

FIG. 6 is a diagram illustrating an example of audience information.
In the audience information 352, as information on the audience, “audience seat”, “audience camera”, “ceiling camera”, “audience seat position”, “sitting state”, “viewing object”, “pupil information”, “precursor detection” Is included.

  In the “audience seat”, a seat identification number for identifying the seat 120 where the audience 130 sits is registered as the identification information of the audience 130. In the “passenger camera”, a passenger seat camera identification number for identifying the passenger seat cameras 221 and 222 for photographing the audience in the audience seat is registered in association with the passenger seat identification number. In the “ceiling camera”, a ceiling camera identification number for identifying the ceiling cameras 231 and 232 for photographing the spectator seat is registered in association with the seat identification number. In the “audience seat position”, the position of the audience seat is registered in association with a seat identification number in two-dimensional coordinates. In the “sitting state”, information indicating whether or not there is a spectator in the corresponding spectator seat is registered. “Seated” indicates that there is a spectator, and “vacant seat” indicates that there is no spectator. In the “visual target”, the game field that the spectator is viewing is registered in association with the audience identification number. The visual target area detected by the spectator visual target detection unit 340 is registered in the visual target. For example, “1” indicates that the audience at the corresponding spectator seat is viewing the first game field 101. In the “pupil information”, information indicating the size of the pupil of the audience 130 is registered in association with the seat identification number. The size of the pupil is detected by the precursor motion detection unit 310 by analyzing the face image data of the audience 130. In “precursor detection”, whether or not a precursor motion has been detected is registered in association with a passenger seat identification number. In the precursor action, a result of the presence / absence of the precursor action detected by the precursor action detection unit 310 is registered.

FIG. 7 is a diagram showing an outline of the audience visual object detection process.
In the example shown in FIG. 7, a spectator 132 seated in the spectator seat 122 and a spectator 133 seated in the spectator seat 123 are viewing the fourth game field 104. The fourth game field 104 is an area surrounded by points represented by position coordinates (X41, Y41), (X42, Y42), (X43, Y43), (X44, Y44). The position coordinates of the spectator seat 122 are (sx4, sy4), and the position coordinates of the spectator seat 123 are (sx5, sy5).

  The spectator visual target detection unit 340 acquires the viewpoint position coordinates (sx4, sy4) of the spectator 132 of the spectator seat 122 and the viewpoint position coordinates (sx5, sy5) of the spectator 133 of the spectator seat 123 from the audience seating position detection unit 330. To do. Also, from the audience viewing direction detection unit 320, θ2 is obtained as the viewing direction 142 of the audience 132, and −θ3 is obtained as the viewing direction 143 of the audience 133.

  Based on this information, as shown in FIG. 7, whether or not the line of sight when the spectator 132 sees the viewing direction 142 from the viewpoint position coordinates (sx4, sy4) passes through the area of the fourth game field 104. Can be evaluated. Similarly, it can be evaluated whether or not the line of sight when the spectator 133 views the viewing direction 143 from the viewpoint position coordinates (sx5, sy5) passes through the area of the fourth game field 104. The spectator visual object detection unit 340 detects a game field visually observed by the spectators 132 and 133 based on the evaluation result.

Hereinafter, processing of the image analysis apparatus 300 will be described in order.
FIG. 8 is a flowchart showing the procedure of the audience visual object detection process.
The image analysis apparatus 300 performs a spectator visual target detection process for detecting a target region that is viewed by the spectator 130 at a predetermined cycle. At the time when the image analysis apparatus 300 starts the audience visual object detection process, the face image data of the audience 130 photographed by the audience cameras 221 and 222 and the audience image data photographed by the ceiling cameras 231 and 232 are updated. Suppose that

[Step S01] The image analysis apparatus 300 acquires the passenger seat image data captured by the ceiling cameras 231 and 232.
[Step S02] The audience seating position detector 330 detects the audience seating position based on the acquired audience image data and the audience seat position (position coordinates) registered in the audience information 352. The spectator seating position detection unit 330 analyzes, for each spectator seat, the occupant seat image data of the ceiling camera that captured the spectator seat, and determines whether the spectator is seated in the spectator seat. When it is determined that the audience is seated, the audience seating position detection unit 330 registers “sitting” in the “sitting state” of the audience information 352. When it is determined that the audience is not seated, the audience seating position detection unit 330 registers “vacant seat” in the “seat state” of the audience information 352.

[Step S03] The audience viewing direction detection unit 320 acquires the face image data of the audience 130 photographed by the audience cameras 221 and 222.
[Step S04] The audience viewing direction detection unit 320 extracts one face image data of the audience 130 that has not been subjected to the process of detecting the audience viewing direction.

  [Step S05] The audience viewing direction detection unit 320 analyzes the extracted face image data of the audience 130, and performs a viewing target area detection process for the audience 130. Details of the visual target area detection processing will be described with reference to FIG.

  [Step S06] The precursor motion detection unit 310 analyzes the extracted facial image data of the spectator 130, and detects whether or not the precursor motion that appears when the spectator 130 visually observes the event of interest is seen. Perform motion detection processing. Details of the precursor motion detection process will be described with reference to FIG.

  [Step S07] The image analysis apparatus 300 determines whether or not the analysis of the face image data has been completed for all the audiences. When the analysis of all the spectators has been completed, the image analysis device 300 advances the process to step S08. If there is a spectator who has not finished the analysis, the process proceeds to step S04.

  [Step S08] The attention level calculation unit 360 statistically processes the visual target areas detected for all the spectators, and calculates the attention level of each game field. Details of the attention level calculation processing will be described with reference to FIG. After the attention level calculation process ends, the game field with the highest level of attention is notified to the output video creation device 400, and the process ends.

FIG. 9 is a flowchart illustrating the procedure of the visual target area detection process.
[Step S51] The audience viewing direction detection unit 320 analyzes the extracted face image data of the audience 130, detects the face direction and eye direction of the audience 130, and detects the viewing direction viewed by the audience 130. The audience viewing direction detection unit 320 detects the viewing direction of the audience 130 in consideration of the positions of the audience cameras 221 and 222 that have taken images and the positional relationship of the audience 130.

  [Step S52] The audience visual object detection unit 340 acquires the viewpoint position coordinates of the audience 130 based on the audience seat position (position coordinates) of the audience seat 120 where the audience 130 sits from the audience information 352. Here, the audience seat position of the audience seat 120 is regarded as the viewpoint position coordinates of the audience 130.

  [Step S53] The spectator visual target detection unit 340 sets the game field k to be determined as 1. Accordingly, it is determined in order from the first game field 101 whether or not the game field corresponds to the viewing target area of the spectator 130.

  [Step S54] The spectator visual object detection unit 340 reads the position (position coordinates) of the game field k from the game hall information 351. The spectator visual target detection unit 340 collates whether or not the direction viewed by the spectator 130 based on the viewpoint position coordinates and the viewing direction of the spectator 130 overlaps the area indicated by the position coordinates of the game field k.

  [Step S55] The spectator visual target detection unit 340 determines whether or not the visual target area of the spectator 130 is the game field k based on the result of the collation. When the viewing direction of the spectator 130 and the area of the game field k overlap, the spectator visual object detection unit 340 determines that the visual object area of the spectator 130 is the game field k. When the viewing direction of the spectator 130 and the area of the game field k do not overlap, it is determined that the visual target area of the spectator 130 is not the game field k. The spectator visual target detection unit 340 advances the process to step S56 when the game field k is the visual target area of the spectator 130, and advances the process to step S57 when it is not the visual target area.

  [Step S56] The spectator visual target detection unit 340 registers the game field k in the spectator information 352 as the visual target area of the spectator 130. The spectator visual target detection unit 340 registers the game field k in the visual target column corresponding to the audience identification number of the spectator seat 120 where the spectator 130 is seated. The audience visual object detection unit 340 ends the processing after registering the visual object area.

  [Step S57] The spectator visual target detection unit 340 determines whether or not collation has been completed for all the game fields. The spectator visual object detection unit 340 advances the process to step S59 when collation is completed for all the game fields. If verification has not been completed for all the game fields, the process proceeds to step S58.

[Step S58] The spectator visual object detection unit 340 advances the identification number of the game field k to be determined by one, and advances the process to step S54.
[Step S <b> 59] The audience visual object detection unit 340 registers the absence in the visual object column corresponding to the audience identification number of the audience seat 120 on which the audience 130 is seated, assuming that the audience 130 does not have an area to be viewed. Thereafter, the process ends.

FIG. 10 is a flowchart showing the procedure of the precursor motion detection process.
[Step S61] The precursor motion detection unit 310 analyzes the extracted face image data of the spectator 130, and detects the size of the pupil of the spectator 130.

  [Step S62] The precursor motion detection unit 310 reads the size of the pupil of the audience 130 measured last time from the audience information 352 based on the audience identification number of the audience seat 120 where the audience 130 is seated. The precursor motion detection unit 310 calculates the difference between the previous and current pupil sizes, and determines whether there is a change in the pupil size of the audience 130. For example, it is determined that there is a change when there is a difference larger than a predetermined size. In addition, for example, a condition that is considered to be changed may be set when the pupil is changed so as to become larger. Note that the size of the pupil of the audience information 352 is updated to the current measurement value. The sign motion detection unit 310 advances the process to step S63 when there is a change in the size of the pupil of the spectator 130, and advances the process to step S64 when there is no change.

  [Step S63] The precursor motion detection unit 310 sets “presence” to “detection of precursor” of the audience information 352 corresponding to the audience identification number of the audience seat 120 of the audience 130 that has detected a change in the size of the pupil, and performs processing. Exit.

  [Step S64] The precursor motion detection unit 310 sets “none” in “detection of precursor” of the audience information 352 corresponding to the audience identification number of the audience seat 120 of the audience 130 that has not detected a change in the size of the pupil. The process is terminated.

FIG. 11 is a flowchart illustrating a procedure of attention level calculation processing. In the example of FIG. 11, the attention level of each game field is represented by the number of spectators who are viewing this game field.
[Step S81] The attention level calculation unit 360 reads the audience information 352 in the order of the audience identification number of the audience seat 120, performs statistical processing, and calculates the attention level of the game field. The attention level calculation unit 360 sets 1 for the spectator seat number m.

  [Step S82] Based on the “sitting state” of the spectator information 352, the attention level calculation unit 360 determines whether or not the spectator 130 is seated in the spectator seat with the seat identification number m (hereinafter referred to as spectator seat m). judge. If the spectator 130 is seated, the process proceeds to step S83. If the spectator 130 is not seated, the process proceeds to step S86.

  [Step S <b> 83] The attention level calculation unit 360 determines whether or not a precursor has been detected in the spectator 130 of the spectator seat m based on “detection of a precursor” in the spectator information 352. The attention level calculation unit 360 proceeds to step S84 when a precursor is detected in the spectator 130 of the audience seat m, and proceeds to step S85 when no precursor is detected in the spectator 130.

  [Step S <b> 84] The attention level calculation unit 360 adds “2” to the attention level of the corresponding visual target area because the sign is detected in the audience 130. It is assumed that the attention level is stored in the storage unit 350 in association with the game field in the visual target area. The attention level calculation unit 360 weights the attention level corresponding to the spectator 130 in which the precursor is detected to be heavier than the attention level corresponding to the spectator 130 in which the precursor is not detected, and the process proceeds to step S86. Note that “+2” is an example, and what value is weighted is appropriately set.

[Step S85] Since no sign is detected in the spectator 130, the attention level calculation unit 360 adds “1” to the attention level of the corresponding visual target area, and the process proceeds to step S86.
[Step S86] The attention level calculation unit 360 determines whether or not the processing for all spectator seats has been completed. The attention level calculation unit 360 advances the process to step S88 when the process for all spectator seats is completed, and advances the process to step S87 when the process is not completed.

[Step S87] The attention level calculation unit 360 adds “1” to the spectator seat m and advances the process to Step S82.
[Step S88] The attention level calculation unit 360 selects a display target to be displayed on the monitor device 500 based on the attention level of each game field calculated from the visual target area of all the spectators. The attention level calculation unit 360 selects the video signal of the game field with the highest level of attention, that is, the game field viewed by the most spectators, and sets it as a display target.

  The video display system according to the second embodiment is most noticed by analyzing the spectator face image data and the spectator seat image data to detect the game field being viewed by the spectator and performing statistical processing. You can select a matchground. Since the video signal of the game field selected in this way is displayed on the monitor device, it is possible to display the video of the game field that attracts the most audience.

  In addition, by detecting the sign motion of the spectator and weighting the attention level of the game field that the spectator who has detected the sign motion is observing, the decisive moment of the game noticed by a part of the spectator is displayed on the monitor screen, Can be provided to other audiences. In sports watching, if a spectator's visual target area is statistically processed and displayed after many spectators view, there is a possibility that a decisive moment noticed by some spectators cannot be displayed. According to the video display system of the second embodiment, a decisive moment can be displayed on the monitor screen by detecting the sign motion of the audience and reflecting it in the display.

[Third Embodiment]
FIG. 12 is a diagram illustrating a configuration of a video display system according to the third embodiment. The same components as those in the video display system according to the second embodiment shown in FIG.

  In the video display system of the third embodiment, the passenger seat cameras 221 and 222 of the second embodiment are replaced with a rotary passenger seat camera 240. In addition, the image analysis device 380 that analyzes the face image data captured by the rotary passenger seat camera 240 has the control function of the rotary passenger seat camera 240 added to the image analysis device 300 of the second embodiment.

  The rotary audience camera 240 can rotate the imaging unit so that all the audience seats 120 can be imaged. Such a rotary audience camera 240 can photograph the face of the audience 130 from the front of the audience 130. There is an advantage that the spectators 130 of all the spectator seats 120 can be photographed by one rotary spectator camera 240 without preparing a plurality of spectator cameras 221 and 222. Further, it is not necessary to correct the face image of the audience 130 based on the positional relationship between the audience 130 and the audience seat cameras 221 and 222.

  FIG. 13 is a block diagram illustrating a functional configuration of the video display system according to the third embodiment. The same components as those in the video display system according to the second embodiment shown in FIG.

  In the image analysis device 380 of the video display system according to the third embodiment, a passenger seat camera control unit 370 is added to the image analysis device 300 according to the second embodiment. The passenger camera control unit 370 determines the rotation speed in accordance with the period in which the precursor motion detection unit 310 and the spectator viewing direction detection unit 320 are executed, and instructs the rotary passenger camera 240. In addition, the passenger seat camera control unit 370 performs control to stop the rotation of the rotary passenger seat camera 240 during shooting in order to prevent blurring during shooting by the rotary passenger seat camera 240.

  The rotary passenger seat camera 240 includes a shooting direction changing unit 241 and a shooting unit 242. The shooting direction changing unit 241 changes the shooting direction according to an instruction from the passenger seat camera control unit 370. The photographing unit 242 photographs the spectator 130 in accordance with an instruction from the audience seat camera control unit 370. A wide-angle camera may be arranged instead of the rotary passenger seat camera 240.

[Fourth Embodiment]
FIG. 14 is a diagram illustrating an example of a split display screen of the video display system according to the fourth embodiment. In the first to third embodiments, the screen of the game field with the highest degree of attention is displayed on the monitor device 500. However, in the fourth embodiment, the screen displayed on the monitor device 500 is the split display screen 530. .

The split display screen 530 includes a display area A531 having the widest display area, a display area B532 whose display area is narrower than the display area A531, and a display area C533.
The display video creation unit 410 of the output video creation device 400 acquires the attention level of each game field calculated by the attention level calculation unit 360, and determines the layout of the divided display screen 530 according to the attention level.

  The display video creation unit 410 sets the video displayed in the widest display area A531 as the video signal of the game field with the highest degree of attention. The display video creation unit 410 sets the video displayed in the display area B532 as the video signal of the game field having the second highest degree of attention. In addition, the display video creation unit 410 sets the video displayed in the display area C533 as the video signal of the game field having the third highest degree of attention.

  In the fourth embodiment, the areas of the display area A531, the display area B532, and the display area C533 are variably set according to the degree of attention. The display video creation unit 410 includes a video signal attention level (hereinafter referred to as attention level A) in the display area A531, a video signal attention level (hereinafter referred to as attention level B) in the display area B532, and a display area C533. The area of the display area A531 is obtained by the following formula based on the attention level of the video signal (hereinafter referred to as attention level C).

Display area A = [A / (A + B + C)] × 100 [%]
Note that 100% corresponds to the case where the entire display screen is the display area. By calculating the display area A531 in this way, a video with a higher degree of attention can be displayed in a wider display area. The display area B532 and the display area C533 are areas obtained by dividing the remaining area. The areas of the display area B532 and the display area C533 are obtained by the following formula.

Display area B (same for display area C) = [(B + C) / (A + B + C)] × 100 ÷ 2 [%]
The display video creation unit 410 updates the area of the display area at a predetermined cycle. The display video creation unit 410 may update the area of the display area at a predetermined timing, for example, when switching the video signal to be displayed in the display area A531.

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the video display apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic storage device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic storage device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Optical discs include DVD, DVD-RAM, CD-ROM / RW, and the like. Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the program, for example, a portable recording medium such as a DVD or a CD-ROM in which the program is recorded is sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. In addition, each time a program is transferred from a server computer connected via a network, the computer can sequentially execute processing according to the received program.

  Further, at least a part of the above processing functions can be realized by an electronic circuit such as a DSP (Digital Signal Processor), an ASIC, or a PLD (Programmable Logic Device).

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, all the constituent elements shown in the embodiment may be appropriately combined. Furthermore, constituent elements over different embodiments may be appropriately combined. Various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1,1a, 1b Game field 2 Audience seat area 3 Audience 3e Viewing direction 10 Video display apparatus 11 Visual object detection part 12 Display control part 13 Display part 13a Monitor screen 31, 32 1st imaging device 40 2nd imaging device 50 Position detection apparatus

Claims (7)

A plurality of first imaging devices that are arranged in each of a plurality of target areas and shoot the arranged target areas;
A second imaging device that captures a face image of a spectator viewing the plurality of target areas;
A position detection device for detecting the viewpoint position of the audience;
Detecting the viewing direction of the audience based on the face image of the audience, based on the viewing direction of the audience, the viewpoint position information of the audience detected by the position detection device, and the position information of the plurality of target areas, A visual target detection unit that detects a visual target area that the spectator is viewing from among the plurality of target areas, and among the plurality of first imaging devices, imaging of the first imaging device corresponding to the visual target area. A video display device comprising: a display control unit that displays the video that has been displayed on the display unit;
A video display system. The visual target detection unit calculates a degree of attention according to the number of spectators viewing the visual target area for each visual target area based on the visual target areas detected for the plurality of spectators, and based on the attention level Selecting the visual target area to be displayed on the display unit,
The video display system according to claim 1. The video display device
An operation detection unit for detecting an operation when the audience is visually interested in an event,
The visual object detection unit performs weighting such that the degree of attention of the visual target area viewed by the audience in which the motion is detected is higher than the degree of attention of the visual target area observed by the audience in which the motion is not detected. Calculating the degree of attention;
The video display system according to claim 2. The motion detection unit detects the size of the pupil of the spectator based on the face image, compares with the size of the pupil detected in the past for the spectator stored in the storage unit, and changes to the size of the pupil When there is, it is presumed that the audience visually observed an event that attracts interest.
The video display system according to claim 3. In the position detection device, the third imaging device acquires a seat image obtained by photographing a spectator seat whose position is determined in advance from above, and whether or not a spectator is seated in the spectator seat based on the seat image. Determining and generating the viewpoint position information of a spectator seated in the spectator seat based on position coordinates defined in advance for each spectator seat;
The video display system according to claim 1. The second imaging device is a rotary audience camera that photographs a spectator seat provided around the target area, and photographs the audience in the spectator seat from the front by rotating the imaging section.
The video display system according to claim 1. Computer
Images taken by a plurality of first imaging devices that are arranged in each of the target regions and photograph the arranged target regions, and a second imaging device that photographs a face image of a spectator viewing the plurality of target regions. Obtaining the audience face image and the viewpoint position information of the audience detected by the position detection device;
Detecting the viewing direction of the audience based on the face image of the audience, and based on the viewing direction of the audience, the viewpoint position information of the audience, and position information of the plurality of target areas, Among them, the visual target area that the audience is viewing is detected,
Displaying a video captured by the first imaging device corresponding to the viewing target region on the display unit among the plurality of first imaging devices;
A video display method for executing processing.

Images