JP2023097056A - Event management server system and content image control method - Google Patents

Abstract

To provide an event management server system for enabling an event visitor to easily use an appropriate service through a portable terminal, and a content image control method to be achieved by the event management server system.SOLUTION: In an event management server system 1, when portable terminals 12a to c are held over a scene at an event site by users visiting the event, their picked-up images are acquired and displayed as a real time video. An event management server 10 acquires the picked-up images from the portable terminals 12a to c and acquires characteristics of the users by collating the picked-up images with registration data. Then, when a player to watch is determined to have left the users by more than a threshold value, an enlarged image of the player is superimposedly displayed on the portable terminals 12a to c.SELECTED DRAWING: Figure 1

Description

The present invention relates to an event management server system that provides services via mobile terminals at event venues such as sports competitions, and a content image control method realized thereby.

At event venues, tourist facilities, and the like, technologies that improve convenience by providing various types of information via mobile terminals held by visitors have been put to practical use. For example, there has been proposed a technique for notifying the smartphones of visitors of the availability of restrooms in a stadium and the locations of vendors who sell food and drinks (see, for example, Patent Document 1). Also, a technique has been proposed in which auxiliary information about a player of interest is displayed using a see-through display or the like while watching a baseball game (see, for example, Patent Document 2). On the other hand, a technology has been proposed in which a match being held at a stadium is captured from multiple viewpoints and images from viewpoints corresponding to requests from clients are distributed (see, for example, Patent Document 3).

Japanese Unexamined Patent Application Publication No. 2018-142080 Japanese Patent Publication No. 2019-531538 JP 2019-68130 A

Most of the above-mentioned services that can be used by event visitors using mobile terminals are relatively simple, such as one-sided presentation of additional information. For this reason, especially at events such as sports competitions and music festivals where the audience's interest tends to focus on performances in the real space, there are few opportunities to dare to look at and operate the terminal, and the situations where the service can be used effectively are limited. was

The present invention has been made in view of these problems, and its object is to provide a technology that enables event visitors to easily use appropriate services via mobile terminals.

One aspect of the present invention relates to an event management server system. This event management server system includes a terminal information acquisition unit that acquires data of an image captured by a terminal held by a user who has visited the venue of the event, and acquires characteristics of the user by comparing registered data with the captured image, A content image control unit that selects a content image to be displayed on a terminal in accordance with the characteristics, and an image data transmission unit that transmits data of the selected content image to the terminal for display. .

Another aspect of the present invention relates to a content image control method. This content image control method comprises steps of acquiring data of an image captured by a terminal held by a user who has visited the venue of the event; Accordingly, the method comprises the steps of selecting a content image to be displayed on the terminal, and transmitting data of the selected content image to the terminal for display.

Any combination of the above constituent elements, and any conversion of the expression of the present invention between a method, an apparatus, a system, a computer program, a recording medium recording a computer program, and the like are also effective as aspects of the present invention. .

According to the present invention, event visitors can easily use appropriate services via mobile terminals.

It is a figure which shows the structure of the event management system which can apply this Embodiment. FIG. 3 is a diagram illustrating a screen displayed by a mobile terminal in the embodiment; FIG. FIG. 10 is a diagram illustrating a screen of a mobile terminal when promoting product sales with a content image in the present embodiment; FIG. 10 is a diagram illustrating a screen of a mobile terminal when enhancing entertainment of an event with a content image in the present embodiment; 3 is a diagram showing an internal circuit configuration of an event management server according to this embodiment; FIG. 1 is a diagram showing an internal circuit configuration of a mobile terminal according to this embodiment; FIG. 3 is a diagram showing the configuration of functional blocks of an event management server according to this embodiment; FIG. 3 is a diagram showing the configuration of functional blocks of the mobile terminal according to the present embodiment; FIG. FIG. 7 is a diagram for explaining an example of matching processing performed by a matching unit of a content image control unit in the event management server according to the embodiment; FIG. 4 is a diagram for explaining an example of a content image displayed on a mobile terminal by a content image control unit of an event management server in the present embodiment; FIG. 10 is a diagram for explaining another example of a content image displayed on a mobile terminal by the content image control unit of the event management server in the present embodiment; FIG. 10 is a diagram showing an example of settings of types of content images and display triggers to be displayed on the mobile terminal by the content image control unit of the event management server in the present embodiment; 4 is a flow chart showing a processing procedure in which an event management server selects and transmits a content image to be displayed on a mobile terminal in this embodiment.

This embodiment relates to a system for providing services to visitors via mobile terminals at an event site. Here, an event may be anything that a plurality of people go to a venue to see and experience, such as a sports competition, music festival, concert, or show. It is not particularly limited. FIG. 1 shows the configuration of an event management system to which this embodiment can be applied. The event management system 1 includes an event management server 10, mobile terminals 12a to 12e, a virtual store management server 14, and a physical store management server 16. FIG.

The mobile terminals 12a to 12e are terminals carried by event attendees (hereinafter referred to as "users"), and display various information related to the event and receive user operations. In FIG. 1, mobile terminals 12a, 12b, and 12c are high-performance mobile phones (smartphones), and mobile terminals 12d and 12e are wearable terminals such as AR (Augmented Reality) glasses. As described above, the mobile terminals 12a to 12e are not limited in form as long as they are general terminals having a display, an information processing device including a communication function, and an input device. Hereinafter, the mobile terminals 12a to 12e will simply be referred to as "mobile terminals 12".

The event management server 10 establishes communication with the mobile terminal 12, receives the captured image and the content of the user's operation, performs corresponding information processing, and returns necessary information. In recent years, with the introduction of 5G (fifth generation mobile communication system), etc., it has become possible to transmit various data with low delay and large capacity even with mobile terminals. Therefore, such technology is appropriately used for communication between the mobile terminal 12 and the event management server 10 as well.

Note that the event management server 10 may actually be realized by a plurality of information processing devices and storage devices. For example, the event management servers 10 may be distributed according to their functions by MEC (Multi-access Edge Computing), which is one of 5G technologies. In this case, functions that require high responsiveness, such as sending and receiving image data to and from the mobile terminal 12, are implemented by a server installed near the event venue, and other functions are implemented by a server installed in a separate data center. Realization is conceivable. Therefore, the "event management server" in this embodiment can be read as an "event management server system" regardless of the number of devices.

The virtual store management server 14 establishes communication with the event management server 10 and mediates trading between the virtual store on the network and the user. The physical store management server 16 establishes communication with the event management server 10 and mediates transactions between users and physical stores inside and outside the event venue. The physical store management server 16 may further provide the event management server 10 with information on the real-time status of the physical store, such as the degree of congestion, waiting time, product inventory, and the like, at any time.

In this embodiment, the event management server 10 provides various services to users via the mobile terminal 12 . The services provided here are broadly divided into mechanisms that make the event itself more enjoyable, and sales promotion of related products. In this embodiment, it is possible to visually recognize various information related to the service together with the state of the event via the portable terminal 12 . This allows users to take advantage of a variety of services without missing out on the real-time status of the event or losing interest.

Specifically, the user captures the state of the event with the in-camera by holding the portable terminal 12, and the real-time image is displayed on the display. On the other hand, the event management server 10 collects images captured by the mobile terminal 12 and analyzes them to estimate the characteristics and interests of each user who handles the mobile terminal 12 . Then, based on the estimation result, an optimal image to be superimposed on the real-time video is generated and displayed on each portable terminal 12 .

The image to be superimposed and displayed here may be represented as a virtual object drawn as if it exists at the event site. Techniques such as AR and MR (Mixed Reality) can be applied as a technique for fusing a virtual object with a real space image. In this case, three-dimensional data representing the spatial structure of the event venue is used to define the placement of the virtual objects. In the present embodiment, the three-dimensional data is also used for analysis of captured images, thereby making it possible to identify user's characteristics and interests at low cost. Hereinafter, an image selected by the event management server 10 for each mobile terminal 12 and superimposed on the real-time video will be referred to as a "content image".

FIG. 2 illustrates a screen displayed by the mobile terminal 12. As shown in FIG. In this example and FIGS. 3 and 4 described later, it is assumed that the event is a soccer game and the user is watching the game on the field from his/her seat. When the user holds up the mobile terminal 12 in this state, the mobile terminal 12 displays the appearance of the real space seen from that viewpoint, as shown in (a), for example. This situation is realized, for example, by immediately displaying a real-time image captured by a camera mounted on the back surface of the mobile terminal 12 on the display. If the user changes the orientation of the mobile terminal 12, the field of view of the image is naturally linked to it.

(b) shows a state in which a content image is superimposed on the real-time video shown in (a). In this example, a cylindrical virtual signage image 200 is superimposed and displayed as the content image. The virtual signage is an object that imitates an electronic bulletin board that displays various information such as scores, images of players, and advertisements on its sides and inner walls, making it appear as if it is hanging from the sky. The virtual signage image 200 is drawn by mapping necessary information as textures on a three-dimensional model representing the shape of the signage. However, this is not intended to limit the type, shape, or drawing method of the object represented by the content image.

The virtual signage image 200 may be animated, and the information to be presented may be circulated on the side surface of the cylinder. When implementing such an AR, the event management server 10 stores in advance three-dimensional data of the spatial structure of the soccer stadium, which is the venue for the event. Then, the placement of the virtual signage is determined for the same three-dimensional space, and the position information is notified to the mobile terminal 12 . The mobile terminal 12 virtually arranges the signage object at the corresponding position in the real space and projects it onto the screen coordinate system of the camera to draw the image 200 on the real-time video being displayed.

As a result, the virtual signage is fixed at a predetermined position in the real space regardless of the field of view of the camera, and content images can be displayed while avoiding important areas of the event such as soccer fields. By converting the 3D data of the spatial structure into a highly accurate data format such as point cloud data, content images can be displayed with high accuracy according to various locations, such as a virtual logo on the underside of the roof of a soccer stadium. can be displayed.

Here, when the user instructs any of the information displayed in the image 200 of the virtual signage, the event management server 10 acquires the user's operation and displays a content image representing other information related to the instructed information. , may be further displayed on the mobile terminal 12 . For example, when the sponsor's logo displayed on the virtual signage is instructed, the event management server 10 additionally displays the advertising video of the sponsor. Also, when the player's face image is designated, an image representing the player's profile, battle record, related goods, etc. is additionally displayed.

A similar display may be triggered by an instruction to the image of the real object represented in the real-time video. For example, when the user points to the signboard image 201 installed in the audience seat on the opposite side, the event management server 10 displays a promotional video associated with it. In addition, the event management server 10 may display a GUI (Graphical User Interface) for accepting the purchase of related products in accordance with the user's instruction operation on the virtual signage image 200 or the image of the real object.

The content image may be always displayed while the user holds the mobile terminal 12 over the event site, or may be switched between display and non-display according to a specific operation by the user as described above. As the specific operation, in addition to a general instruction operation represented by a tap operation on the screen, a photographing operation such as a zoom magnification changing operation or a still image photographing operation may be used. Alternatively, display/non-display may be switched based on a factor other than user operation, such as the state of the contestants or the situation of the event.

FIG. 3 exemplifies the screen of the mobile terminal 12 when promoting product sales with content images. In (a), sale information 206 of spikes, which is a product, and a GUI 208 for accepting purchases are superimposed on a real-time video. For example, when the user indicates an image of a player displayed on the virtual signage image 200 shown in FIG. , the sales information 206 and the GUI 208 are displayed. These content images may be objects having three-dimensional models, or planar images.

For the image of the spike included in the sales information 206, MR technology may be used to change the direction of the three-dimensional object and draw it towards you, so that you can check the details. Also, in the illustrated example, the GUI 208 is represented by a horizontally elongated three-dimensional object representing the characters "buy this". When a user operation instructing the GUI 208 is performed, the event management server 10 accepts it, and executes order processing and payment processing to the virtual store management server 14 or the real store management server 16 that manages the seller.

When the ordering and payment processing is completed, the event management server 10 superimposes and displays an image 210 indicating purchase completion on the mobile terminal 12 as illustrated in (b). In this example, the image 210 is represented by a horizontally elongated three-dimensional object representing the text "purchase completed". Products purchased in this manner may be available at the venue or may be delivered at a later date, depending on the type and place of purchase. In the former case, the package may be delivered to the seat of the visitor, or the visitor may go to the store to pick it up.

FIG. 4 illustrates the screen of the mobile terminal 12 when enhancing the entertainment of the event with content images. (a) is an example of the screen when a goal is scored. Information 220 of the player who made the shot and an image 222 representing a virtual effect of lights and characters celebrating the score are superimposed on the real-time video as content images. it's shown. (b) superimposes a real-time image 224 of a player on the field as a content image at an arbitrary timing during the game. By enlarging and displaying the image 224 of the player as shown in the figure, it is possible to confirm in detail the appearance of the player, which is difficult to see from his own seat.

Especially in the case of the content image shown in FIG. 4, whether the user finds the display result suitable or annoying depends largely on the tastes and preferences of individual users, such as the team they are rooting for or the player they are watching. Also, depending on the user's seat and the player's position, it may not be necessary to enlarge the player's image. Furthermore, information desired to be viewed and products desired to be purchased may change from moment to moment due to events that are difficult to predict in advance, such as battle conditions, weather, temperature, conditions of other spectators, physical conditions of individual users, and the like.

Based on this, in the present embodiment, the event management server 10 collects information such as images captured by the mobile terminal 12, details of user operations such as shooting operations and instruction operations, user profiles, etc. Or presume interest. Furthermore, the contents of the content images to be displayed and the display timing are optimized in consideration of the contestants and the situation in the hall. As a result, it is possible to enjoy the event more and easily purchase desired products through the mobile terminal 12 while minimizing the time and effort required for pre-registration and menu operations.

FIG. 5 shows the internal circuit configuration of the event management server 10. As shown in FIG. The event management server 10 includes a CPU (Central Processing Unit) 23 , a GPU (Graphics Processing Unit) 24 and a main memory 26 . These units are interconnected via a bus 30 . An input/output interface 28 is also connected to the bus 30 . The input/output interface 28 includes a communication unit 32 that establishes communication with the mobile terminal 12, the virtual store management server 14, and the real store management server 16, a storage unit 34 such as a hard disk drive and a nonvolatile memory, and a display (not shown). An output unit 36 for outputting, an input unit 38 for inputting data from an input device (not shown), and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.

The CPU 23 controls the entire event management server 10 by executing the operating system stored in the storage unit 34 . The CPU 23 also executes various programs read from the removable recording medium and loaded into the main memory 26 . The GPU 24 performs drawing processing according to the drawing command from the CPU 23 and outputs the result to the output unit 36 . The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing.

FIG. 6 shows the internal circuit configuration of the mobile terminal 12. As shown in FIG. The mobile terminal 12 includes a CPU 50 , a main memory 52 , a display section 54 , an input section 56 , an audio output section 59 , a communication section 60 , a motion sensor 62 , a camera 64 and a position acquisition section 66 . These units are interconnected via a bus 58 . The CPU 50 processes information acquired from each section of the mobile terminal 12 via the bus 58 and supplies output data to the display section 54 and the audio output section 59 . A main memory 52 stores programs and data necessary for processing in the CPU 50 .

The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel and its driving mechanism, and displays images such as those illustrated in FIGS. The input unit 56 is composed of a touch pad that covers the screen of the display unit 54 and hardware keys provided on the housing. accept. The audio output unit 59 is composed of a speaker, earphones, and their control mechanism, and allows the user to listen to audio as necessary.

The communication unit 60 establishes communication with the event management server 10 and transmits and receives necessary data. In this embodiment, mobile communication of the mobile terminal 12 is realized by, for example, 5G communication technology. The motion sensor 62 is composed of at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like, and measures the posture and movement of the mobile terminal 12 in a three-dimensional space. The camera 64 shoots moving images of the real space in the opposite direction to the screen of the display unit 54 . Especially in this embodiment, the image is immediately displayed on the display unit 54 so that the state of the real space can be seen in real time. The captured image data is also transmitted to the event management server 10 via the communication unit 60 and used for selecting content images.

The position acquisition unit 66 acquires the position information of its own device at any time using well-known technologies such as GPS (Global Positioning System) and A-GPS (Assisted-GPS). The values measured by the motion sensor 62 and the position acquisition unit 66 and the image captured by the camera 64 can be used to determine the screen coordinate system for the real space when rendering the virtual object, which is the content image.

FIG. 7 shows the configuration of functional blocks of the event management server 10 in this embodiment. Each functional block shown in FIG. 7 and FIG. 8, which will be described later, can be realized by electronic components such as the CPU and main memory shown in FIG. It is implemented by a program loaded in memory that performs various functions such as a data input function, a data holding function, an image processing function, and a communication function. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to either one.

The event management server 10 includes a terminal information acquisition unit 70 for acquiring captured image data and details of user operations from the mobile terminal 12, a content image control unit 72 for controlling content images to be displayed on the mobile terminal 12, and a product trading procedure. A trading processing unit 74 to implement, an information storage unit 76 for storing information necessary for content image control and product trading procedures, a contestant status information acquiring unit 78 for acquiring predetermined information related to the status of contestants in the event, It has an event status information acquisition section 80 that acquires predetermined information relating to the status of an event, and an image data transmission section 82 that transmits content image data.

The terminal information acquisition unit 70 acquires data of images captured in real time by the camera 64 of the mobile terminal 12 . The terminal information acquisition unit 70 also acquires details of user operations on the mobile terminal 12 at any time. Here, the user operation includes, for example, an operation to change the zoom magnification of the camera 64, a shooting operation such as a still image shooting operation, an instruction operation to an image being displayed on the mobile terminal 12, a product purchase operation, and the like. The terminal information acquisition unit 70 may continuously acquire data of captured images at predetermined time intervals while the mobile terminal 12 is capturing images of the event venue, or may acquire data of captured images at predetermined time intervals. Only, the data of the photographed image at that time may be acquired in association with the content of the operation.

The terminal information acquisition unit 70 acquires the data of the captured image and the information related to the content of the user's operation in association with the identification information of the mobile terminal 12, so that the subsequent processing can be performed individually for each mobile terminal 12. make it possible. The content image control unit 72 appropriately selects and generates content images to be displayed on each mobile terminal 12 for each user based on the data and information acquired by the terminal information acquisition unit 70 . Specifically, the content image control section 72 includes a matching section 90 , an image selection section 92 and an image generation section 94 .

The collation unit 90 collates the data and information acquired by the terminal information acquisition unit 70 with three-dimensional data indicating the spatial structure of the event site, or pre-registered template images such as contestants and logos. The characteristics and interests of the user using the terminal 12 are estimated. For example, the collation unit 90 identifies the position of the mobile terminal 12, and thus the position of the user, by collating the spatial structure data of the event venue with the captured image.

In this case, the "matching" is realized by, for example, searching for three-dimensional data of the spatial structure based on the feature point distribution of the captured image, and deriving the position and orientation of the mobile terminal 12 when the most matching angle of view is obtained. be done. However, the means for specifying the position of the user is not limited to this, and the output values of the motion sensor 62 and the position acquisition unit 66 provided in the mobile terminal 12 are used, or the output values are combined with the results of collation of the captured images to improve accuracy. You can raise it.

The matching unit 90 may specify secondary information using the specified user's position. For example, the collation unit 90 may derive the relation of the distance and direction between the event participant who is the subject and the user. Alternatively, the matching unit 90 may determine the team the user is supporting or estimate a profile according to the rank of the seat by acquiring the type of seat that includes the specified position.

Further, the matching unit 90 may estimate the person or object that the user is paying attention to by matching the image of the subject in the captured image with the image or character information prepared in advance. For example, the matching unit 90 extracts the face area from the photographed image, matches it with the registered face image of the contestant, detects the uniform number and name indicated on the uniform, and compares it with the registered information. By doing so, the contestant of interest is specified. Alternatively, the matching unit 90 detects the signboard or product logo in the captured image and matches it with the logo image of the registered company or product, thereby specifying the company or product that the user is interested in at that time. may Techniques used in fields such as object detection and object recognition can be used for such "collation" between images.

The timing at which the collation unit 90 performs collation as described above is not limited. For example, the matching unit 90 may perform the matching only once at the start of the event, or may perform the matching periodically while the captured image data is being transmitted from the mobile terminal 12 . Alternatively, the collation unit 90 may collate when the situation of the event, the state of the contestants, the content of the user's operation, and the like satisfy predetermined conditions. For example, when the user performs a zoom-in operation, the matching unit 90 may set the image of the subject at the center of the zoom as a matching target.

The image selection unit 92 selects content images to be displayed on the mobile terminal 12 based on the result of matching by the matching unit 90 . For example, the image selection unit 92 selects the content image representing the virtual effect shown in FIG. 4(a) when the team that the user supports scores. Also, when the contestant being watched is in a place away from the user, an enlarged image as shown in FIG. 4(b) is selected. Alternatively, when a billboard in the hall is zoomed in, an advertisement for a related company or product, or a GUI for purchasing related products as shown in FIG. 3 is selected as the content image.

The image generation unit 94 generates the content image selected by the image selection unit 92 in a displayable state on the mobile terminal 12 . When a content image is represented by a three-dimensional object fused with the real space, the image generator 94 associates the model data of the three-dimensional object with the position in the three-dimensional coordinate system representing the event site. Furthermore, the image generator 94 may generate texture data to be mapped onto a three-dimensional object based on real-time information, or cut out a real-time image of a specific contestant from a live-action video. Alternatively, the image generator 94 may render the content image as a two-dimensional image.

The contestant state information acquisition unit 78 continuously acquires state information including at least one of the positions and orientations of event contestants. For example, the contestant state information acquisition unit 78 acquires real-time images of the contestants photographed from a plurality of viewpoints from an external camera (not shown), and analyzes them to determine the positions, orientations, and speeds of the contestants at the event venue. , to derive state information such as the type of action. Such image analysis is widely put into practical use in fields such as sports broadcasting.

The contestant status information acquisition unit 78 sequentially provides the content image control unit 72 with the contestant status information and video data. The content image control unit 72 uses the information and data to control the content image. For example, the content image control unit 72 determines whether to display an enlarged image of the contestant according to the distance between the user and the contestant. Alternatively, the content image control unit 72 changes at least one of the position and direction of the virtual viewpoint when generating the content image so as to represent the appearance of the contestants who are oriented so that they cannot be seen by the user.

As a result, even if it is difficult to see the contestants in a large venue or the like, the visibility can be improved regardless of the seats. Further, the content image control unit 72 may perform control so that, according to the performance of the contestant, a virtual effect emphasizing it or commentary on the performance is displayed.

The event status information acquisition unit 80 acquires event status information such as weather, time of day, season, temperature, humidity, match status, and show progress. The event status information may be input by the administrator of the event management server 10 or the organizer of the event before or during the event, if necessary. You may specify from the output value by various sensors which are not illustrated. Alternatively, the event situation information acquisition unit 80 may acquire situation information by analyzing a captured image transmitted from the mobile terminal 12 or a real-time video captured by an external camera (not shown).

The event status information acquisition unit 80 appropriately provides event status information to the content image control unit 72 . The content image control unit 72 uses the information to control the content image. For example, the content image control unit 72 displays a content image for accepting the purchase of rain gear when it rains during the event, or displays a content image for accepting the purchase of drinking water when the temperature rises.

The content image control unit 72 also causes the portable terminal 12 of the user who is cheering for the team that has scored a score to display a virtual effect such as that shown in (a) of FIG. A content image showing information on the scorer may be displayed. Furthermore, the content image control unit 72 may display discount coupons that can be used to purchase related products on the portable terminal 12 of the user who is cheering for the team that scored. In this case, the discount rate may be changed according to the battle situation, such as increasing the discount rate as the score increases.

Also, the collation unit 90 of the content image control unit 72 may switch the spatial structure data to be collated with the captured image based on the situation information. For example, in the case of a stadium with a roof that can be opened and closed, the state of the roof changes depending on the weather. For this reason, spatial structure data is prepared in a plurality of patterns with different roof conditions, and the collation unit 90 selects data representing the corresponding roof condition based on the weather included in the situation information and performs collation. use. As a result, the accuracy of matching results can be stabilized. Note that the event situation information acquisition unit 80 may acquire the state of the spatial structure itself, such as whether the roof is closed or open, as situation information from a captured image or the like.

Further, when the content image is represented by AR, the switching of the data of the spatial structure based on the situation information may be reflected even at the stage where the image selection unit 92 selects the content image. As a result, it is possible to avoid the problem of displaying a virtual logo as a content image at a position behind the roof that does not actually exist. The contestant status information and the event status information described above are used for content selection in the content image control unit 72, and may also be used as a trigger for displaying content images.

The trading processing unit 74 implements a trading procedure according to a purchase operation on a GUI for purchasing a product displayed as a content image on the mobile terminal 12 . At this time, the trading processing unit 74 acquires pre-registered personal information of the user of the portable terminal 12, thereby saving the user the trouble of re-entering the personal information on the spot. Here, personal information includes information necessary for settlement such as name, credit card number, security code, and information such as delivery address and telephone number in the case of mailing the product.

As described above, the product to be purchased may be an item received by the user when the user goes to the store in the hall, or may be an item delivered from the store to the user's seat. In the latter case, the transaction processing unit 74 may refer to the user's seat information acquired by the collation unit 90 as a result of collation to identify the delivery destination. In any case, the trading processing unit 74 establishes communication with the virtual store management server 14 and the real store management server 16 according to the user's purchase operation, and executes order processing and payment processing. Conventional technology can be applied to the mechanism of electronic commerce using payment means such as credit cards and electronic money.

If the purchase operation is performed without designating a store, or if the target product is sold at a plurality of chain stores, the transaction processing unit 74 uses the user's position and A store closest to the seat may be selected as an order destination. Further, after placing an order, the sales processing unit 74 may display a content image showing a notification that the order has been completed on the mobile terminal 12 of the purchaser via the content image control unit 72 . By these procedures, the user can purchase the desired product with a simple operation and with a minimum concealment range of the real-time video of the event, as shown in FIG.

After placing an order to a store in the venue, the trading processing unit 74 acquires the location of the store, the order number, the waiting time, etc. The image may be displayed on the portable terminal 12 of the purchase source. In addition, the sales processing unit 74 acquires store status information such as the degree of congestion, product inventory, waiting time, etc. of physical stores in the venue and nearby stores at any time, and transmits content images representing such information via the content image control unit 72. may be displayed on the mobile terminal 12.

The information storage unit 76 stores information necessary for the content image control unit 72 and the trading processing unit 74 to carry out the processes described above. Specifically, information storage unit 76 includes user information storage unit 100 , related information storage unit 102 , spatial information storage unit 104 , matching rule storage unit 106 , and image data storage unit 108 . The user information storage unit 100 associates and stores the identification information of the mobile terminal 12 and the user's personal information. The personal information is used by the trading processing unit 74 in the merchandise trading procedure as described above, and can also be used by the content image control unit 72 when selecting a content image.

For example, the user information storage unit 100 stores at least one of the user's age, gender, address, purchase history, event participation history, etc. as personal information. By referring to the information, the content image control unit 72 optimizes the product to be displayed as the content image, and integrates it with the collation result of the collation unit 90 to improve the accuracy of specifying the user's characteristics and the target of attention. can. For example, when a plurality of contestants and products are shown in the photographed image, the verification unit 90 may narrow down which contestants and products the contestants are paying attention to based on the personal information as described above.

The related information storage unit 102 stores template data used for matching by the matching unit 90 in association with subject information represented by the template data. The template data used for matching includes, for example, images of contestants' faces, uniforms and costumes, team logos, character information indicating team names, images of equipment such as balls, images of signboards, logos of products and companies, Mark, etc.

Each image or text information is associated with the profile of the contestant or team represented by the image, the profile of the company, the sales information of the product, or the like. The matching unit 90 estimates the contestants, companies, and products that are of interest to the user based on the results of matching between the photographed image and the template data. Accordingly, the image selection unit 92 can appropriately select contestants, advertisements, sales information, and the like to be displayed as content images.

The related information storage unit 102 may store the sales information of the product in association with the display condition of the content image. Display conditions can be set for, for example, status information of contestants and status information of an event. As a result, the image selection unit 92 displays product sales information at an appropriate timing without depending on user operation, such as displaying a GUI for purchasing rain gear according to the rainfall, or displaying a discount coupon according to the score. be able to.

The related information storage unit 102 may further store the URL (Uniform Resource Locator) of the store, the location of the actual store, and the like as the sales information of the product. As a result, the trading processing unit 74 can place an order to an appropriate store via the virtual store management server 14 or the real store management server 16 .

The space information storage unit 104 stores data representing the structure of the real space of the event venue as three-dimensional information. Although the data format is not particularly limited, for example, point cloud data (point cloud ) can be used. By using point cloud data, the position and orientation of a virtual object can be defined precisely in the same three-dimensional coordinate system, and the fusion of the real space and the virtual object can be expressed with high precision.

As described above, in the present embodiment, the spatial structure data is also used for the purpose of identifying the position of the user by matching with the captured image. The spatial information storage unit 104 further stores the divided areas of the spatial structure data in association with categories in the real world, such as seats, fields, stages, aisles, and shops. This makes it possible to specify the characteristics of the user according to the seat, or to narrow down the direction in which the camera of the mobile terminal 12 faces.

As described above, when the structure of the event venue is variable, the spatial information storage unit 104 may store a plurality of patterns of spatial structure data, and the matching unit 90 may switch the data used for matching according to a predetermined rule. Therefore, the spatial information storage unit 104 may store the selection criteria set in the situation information or the like in association with each data of the spatial structure.

The matching rule storage unit 106 stores rules for matching processing by the matching unit 90 . For example, as shown in the following table, the matching rule defines targets for matching captured images, triggers for matching, and targets to be specified by matching.

In this example, it is stipulated that the location and seat of the user are identified by matching the photographed image with the spatial structure of the event site when an application is activated on the mobile terminal 12 . In addition, it is stipulated that when performing a zoom-in operation on the mobile terminal 12 and when instructing a display image, the target image and the template image are matched to identify the participant, affiliated company, and product of interest. .

The above table also shows the types of content images selected by the image selection unit 92 based on the specified information. Specifically, by specifying the user's position and seat, and the contestants who are paying attention, the content for the event, that is, the target team that improves visibility or presents a virtual production for the performance and limit the number of participants. In other words, content images of teams and contestants not supported are not displayed, thereby reducing the possibility of making the user feel excessive or uncomfortable.

In addition, by specifying the companies and products of interest, the targets for displaying sales promotion contents, that is, sales information and advertising videos, and accepting purchases are limited. As a result, it is possible to avoid annoyance due to the display of information on companies and products that the customer is not interested in, and increase the sales effect with less information. However, the above table is an example, and is not intended to limit the present embodiment. For example, a content image for purchasing goods related to a contestant of interest can be positioned as sales promotion content.

The image data storage unit 108 stores image data used by the image generation unit 94 to generate content images. The image data storage unit 108 stores, for example, three-dimensional model data for drawing virtual objects, texture data, arrangement information in the three-dimensional space of the event site, and the like. The image data storage unit 108 also stores content image formats showing information about contestants and products, image data of products and advertisements, and the like. These data are associated with the information on contestants, companies, and products stored in the related information storage unit 102 .

The image data transmission unit 82 transmits the content image data selected and generated by the content image control unit 72 to the mobile terminal 12 . Here, the data transmitted by the image data transmission unit 82 may be the content image itself, or may be necessary for the mobile terminal 12 to interpret and draw the image, such as the arrangement of the virtual object in the real space and the character information represented there. It may be partial data.

In the latter case, the portable terminal 12 arranges the object in a coordinate system representing the real space based on the internally held three-dimensional model and the transmitted position information, and then displays the image in the screen coordinate system formed by the display screen. draw. Alternatively, the image data transmission unit 82 may generate an image by making the area other than the content image transparent, and transmit the generated image to the portable terminal 12 . In this case, the mobile terminal 12 only needs to superimpose the transmitted image on the real-time video. It is understood by those skilled in the art that the data held by the event management server 10 and the portable terminal 12 and the sharing of processing may be considered in various ways.

FIG. 8 shows the configuration of functional blocks of the mobile terminal 12 in this embodiment. All or part of the illustrated functions may be configured by downloading from a distribution server that distributes applications according to user selection or autonomous request control of the mobile terminal 12, or may be installed when the mobile terminal 12 is shipped. It may be configured by, for example, The mobile terminal 12 may be composed of a terminal such as a high-performance mobile phone in which an application is installed and a wearable display such as AR glasses for displaying images. In this case, both of them may transmit and receive data by short-range wireless communication such as Bluetooth (registered trademark), or they may be connected by wire.

The portable terminal 12 includes an operation reception unit 110 that receives an operation by a user, a photographing unit 111 that photographs an image of a real space, a terminal information transmission unit 112 that transmits data of the photographed image and details of user operation to the event management server 10, It includes a state information acquisition unit 114 that acquires information related to the state of the terminal 12 itself, an image data reception unit 116 that receives content image data from the event management server 10, and a display unit 118 that displays real-time video and content images.

The mobile terminal 12 further includes a personal information storage unit 120 that stores personal information of the user, and an image data storage unit 122 that stores data used for displaying content images. The operation accepting unit 110 accepts user operations such as an instruction operation and a shooting operation on the image displayed on the display unit 118 . User operations include tapping and pinching on the screen, virtual pointing of virtual objects in real space within the field of view of the camera using MR technology, and hand gestures within the field of view. It's okay.

The photographing unit 111 includes the camera 64 shown in FIG. 5, and photographs an image of the real space where the event is taking place. Especially in the present embodiment, it is basically based on photographing the state of the event while the user is holding the portable terminal 12 over it. The terminal information transmission unit 112 transmits an image captured by the image capturing unit 111, that is, frame data of a moving image, and the content of the user operation received by the operation reception unit 110, together with the identification information of the mobile terminal 12, to the event management server 10. .

Therefore, the personal information storage unit 120 stores at least identification information of the mobile terminal 12 . The personal information storage unit 120 may also store personal information such as the user's name, telephone number, login ID, password, etc., which are recorded in general terminals. State information acquisition unit 114 includes motion sensor 62 and position acquisition unit 66 shown in FIG. 5, and acquires the position, posture, and movement of portable terminal 12 as state information as needed. The state information may be transmitted to the event management server 10 via the terminal information transmission section 112 .

In this case, the event management server 10 may identify the position and orientation of the user with higher accuracy by integrating the matching result using the captured image and the status information of the mobile terminal 12 . The image data receiving unit 116 receives content image data from the event management server 10 . The content image data may be received intermittently once or multiple times as required, or may be received steadily at predetermined time intervals.

The display unit 118 displays the real-time video captured by the image capturing unit 111 and superimposes a content image based on the data received by the image data receiving unit 116 . For example, the display unit 118 projects a three-dimensional model of an object to be displayed as a content image onto a screen coordinate system corresponding to the position and orientation of the mobile terminal 12 acquired by the state information acquisition unit 114, thereby displaying the content image. draw. The display unit 118 may also move or transform the object represented as the content image so that the object interacts with the user's finger or the like.

The image data storage unit 122 stores part of the data necessary for displaying the content image. As described above, the data stored in the image data storage unit 122 may vary depending on the division of processing between the mobile terminal 12 and the event management server 10 . Note that the content image displayed by the display unit 118 is not limited to being fixed to the coordinate system of the real space. For example, an advertisement video of a company, an image for purchasing a product, a discount coupon, or the like may be displayed at a predetermined position in the screen coordinate system depending on the type of image and display timing.

FIG. 9 is a diagram for explaining an example of matching processing performed by the matching unit 90 of the content image control unit 72 in the event management server 10. As shown in FIG. The upper part of (a) shows part of the spatial structure 230 of the soccer stadium, which is the event venue. This data may actually be represented in the form of point cloud data in a three-dimensional coordinate system with axes set at predetermined positions and directions, and is stored in the spatial information storage unit 104 in advance.

When a photographed image 232 is transmitted from the mobile terminal 12, the matching unit 90 extracts feature points from the image, and specifies a range 234 of the spatial structure in which the distribution with the highest matching rate is obtained, as indicated by an arrow. . (b) schematically shows a bird's-eye view of a soccer stadium, and spectator seats 238 are arranged so as to surround a field 236 . Based on the matching result shown in (a), the matching unit 90 specifies the position of the mobile terminal 12 where the field of view of the range 234 is obtained, and thus the user's position 240 .

The matching unit 90 may further identify the seat category corresponding to the position 240 based on the information associated with the spatial structure data. In this example, user position 240 corresponds to a seat behind goalpost 242, typically for supporters of either team. Which team's supporter's seat is is uniquely determined by the setting of the game.

As a result, the collation unit 90 can identify the position of the user and the team that the user supports by collating the photographed image with the spatial structure data. In addition, if there are ranks in the seats, it is possible to estimate the range of expenses that the user expects for the event. Also, if the user is in the special seat, it can be estimated that the user is a high-ranking customer. These user characteristics can be used to determine the teams to be displayed in event content and the price ranges of products to be displayed in sales promotion content.

Note that the collation unit 90 may acquire not only the user's position 240 but also the orientation of the mobile terminal 12 and the line-of-sight direction 244 of the camera by collating the photographed image and the spatial structure data. A method such as Visual SLAM (Simultaneous Localization and Mapping) may be introduced to acquire the position and orientation of the camera from the captured image. By acquiring the line-of-sight direction 244, the content image control unit 72 may estimate a real object such as a contestant existing ahead of the line of sight to be the target of the user's attention and use it for selecting a content image. As a result, for example, by moving the field of view of the camera, it is possible to realize a mode in which the displayed content images are replaced one after another.

FIG. 10 is a diagram for explaining an example of a content image displayed on the mobile terminal 12 by the content image control unit 72 in the event management server 10. As shown in FIG. This example also assumes that the event venue is a soccer stadium, and that the user is in a seat behind one of the goalposts, as shown in FIG. 9(b). (a) and (b) are real-time images displayed on the portable terminal 12 at the seat.

In the case of soccer, the goalposts attacked by each team are switched between the first half and the second half. For this reason, the user behind the goal post has a limited period of time during which he or she can closely watch the team they are cheering for as they attack the goal. For example, in the first half of the game, as shown in (a), the players of the team they are cheering for can be seen relatively close up, while in the second half of the game, as shown in (b), there is a shot at the goalpost on the opposite side. becomes difficult to see. For this reason, it is conceivable that the second half lacks the excitement of cheering compared to the first half.

Therefore, when the distance between the user and the player is greater than or equal to a predetermined value, the image selection unit 92 determines to display the enlarged image 250a of the player image as shown in (c). As described above, the player's distance may be acquired by the contestant state information acquisition unit 78, or may be specified by the image selection unit 92 from the size of the contestant's image in the captured image. Also, the target to be enlarged may be limited to the player of the team that the user supports, which is specified from the user's seat, or the player that the user is paying attention to, which is estimated by a zoom-in operation or the like.

Two or more thresholds may be set for the distance between the user and the player, and the magnification of the enlarged image 250a may be increased step by step each time the distance exceeds the threshold. Alternatively, the magnification of the enlarged image 250a may be continuously increased with increasing distance without providing a threshold. When the distance decreases, the magnification of the enlarged image 250a may be decreased or the enlarged image 250a may be hidden by the opposite control.

The enlarged image 250a shown in (c) is an enlarged image of the player viewed from the same or substantially the same direction as the user's viewpoint. On the other hand, as another aspect, an enlarged image 250b viewed from a direction that cannot be seen from the user's position may be displayed. In this example, by displaying an image of the player viewed from the front, it is possible to check facial expressions that are not normally visible. The user may be allowed to freely operate the direction of the player represented as the enlarged image. Technology that allows players to be viewed from arbitrary viewpoints during play using multi-viewpoint video has been put into practical use, so it may be applied.

Note that an image viewed from a virtual viewpoint different from the user's viewpoint may be of the same magnification. Also, the trigger for display does not have to depend on the distance between the user and the contestants. In any case, at least one of the magnification of the image and the displayed orientation of the contestants is changed according to the relationship between the positions and orientations of the user and the contestants. As a result, for example, at a concert or the like, any performer on the stage can be viewed from any direction, and the difficulty in viewing caused by the angle of the seat can be reduced.

Although only one player is shown in the figure for the sake of clarity, there are, of course, multiple players on the field. Therefore, the target of enlargement may be a plurality of players. In this case, the positional relationship between the players, that is, the formation may be confirmed by adjusting the magnification. Alternatively, the image of the players may be replaced with figures, etc., and the formation may be shown more easily by showing a bird's-eye view. Also, once the player that the user is paying attention to can be specified as described above, information such as the player's battle record and play details may be automatically displayed regardless of whether or not the display is enlarged.

FIG. 11 is a diagram for explaining another example of a content image displayed on the portable terminal 12 by the content image control unit 72 in the event management server 10. As shown in FIG. (a) is a real-time image when the user points the camera at the spectator seats in the soccer stadium. Here, it is assumed that the user focuses on the product 260 held by a certain customer and zooms in on the vicinity thereof as shown in (b).

When the event management server 10 acquires the information about such a shooting operation and the data of the shot image at that time, the matching unit 90 of the content image control unit 72 stores the zoomed-in shot image in the related information storage unit. The images of companies and products registered in 102 are collated. As a result, the matching unit 90 identifies information about the company that manufactures the product 260 and the product itself. In the illustrated example, based on the character string "AA", the color of the rectangle surrounding it, the shape and size of the package, etc., the product 260 is identified as "AA potato" French fries.

In response to this, the image selection unit 92 selects the content image associated with the product. In the illustrated example, an image 262 representing a free coupon for "AA potato" is superimposed. For example, when the user taps the coupon image 262 on the screen of the mobile terminal 12, the mobile terminal 12 stores the image in memory so that it can be displayed again, thereby obtaining the coupon. In this case, when the user goes to the store and presents the coupon, "AA potato" is presented.

Note that the content image represented by photographing the actual object is not limited to the coupon, and may be information on a related company, information on another product sold by the company, GUI for purchase, or the like. A similar content image can also be realized by photographing a signboard posted in the venue as shown in FIG. On the other hand, as shown in FIG. 11, by using products held by other spectators as the basis for displaying content images as recognition targets, diversity can be expanded and unexpected encounters with companies and products can be generated. . In addition, since it is possible to confirm the actual item that a person is eating or wearing, the image is evoked and sales can be further promoted.

In the illustrated example, the user's zoom-in operation is used as a trigger for verification and content image display, but other shooting operations such as zoom-out and still image shooting may be used as triggers. For example, when performing a zoom-out operation, if there is a real object whose entire image has become visible due to the operation, the matching unit 90 may use the image of the real object as a target of matching. Also, when a still image is captured, the collating unit 90 may collate the still image with the registered data in the same manner as when zooming in.

FIG. 12 shows a setting example of the type of content image displayed on the mobile terminal 12 and the display trigger by the content image control unit 72 of the event management server 10 . This setting information is stored in advance in a memory inside the image selection unit 92, for example. As described above, content images are roughly classified into event content and sales promotion content.

The former is an image that improves the user's visibility or evokes a feeling of cheering, and is preferably limited to images related to the team or athlete that the user is cheering for. The latter is an image that promotes sales of various products, and is preferably limited to images that the user is interested in, or that are suitable for the event situation and the user's characteristics. However, like the virtual signage image 200 shown in FIG. 2, depending on the content to be presented, there may be a content image common to all users.

In the example shown in the figure, when a goal is scored in a game such as soccer, the content for the event is set to display a virtual presentation to celebrate and a discount coupon for merchandise related to the team that scored. In addition, when the distance between the user and the contestant is equal to or greater than a threshold value, an enlarged image of the contestant is displayed, and when the user zooms in on the contestant or instructs the contestant, the image can be viewed from a direction different from the user's line of sight. It is set to display an image (another viewpoint image) representing the contestants.

In addition, as sales promotion content, when the user zooms in on signboards and products in the hall, or when environmental changes such as weather and temperature exceed a predetermined value, a GUI is displayed to accept the purchase of the associated product. It is set to let In addition, when the user zooms in on a billboard or product in the hall, it is set to display a discount coupon for the associated product. However, these settings are only an example, and are optimized according to the content of the event, the size and characteristics of the venue, and so on.

Setting items are not limited to those shown in the drawings, such as setting a trigger to hide the displayed content image in addition to the display trigger of the content image. In any case, settings should be made so that appropriate content images can be displayed flexibly according to the circumstances of the event, the relationship between the positions and orientations of the user and the contestants, and the like, not just when the user intentionally instructs. . As a result, the user can naturally raise their mood and easily purchase the desired product without losing their awareness of the event.

Next, operations that can be realized by the above configuration will be described. FIG. 13 is a flow chart showing a processing procedure in which the event management server 10 selects a content image to be displayed on the mobile terminal 12 and transmits it. First, when a user activates an application that realizes display of content images at an event venue and performs authentication such as login, the event management server 10 acquires information input by the user and performs authentication processing (S10). . As a result, the event management server 10 can refer to the user's personal information stored in the user information storage unit 100, and smoothly carry out product sales procedures.

When the authentication is successful, the event management server 10 notifies the mobile terminal 12 of the fact, and the mobile terminal 12 activates the camera and displays a real-time image on the display. Then, the terminal information acquisition unit 70 of the event management server 10 starts acquiring terminal information such as data of the image captured by the camera and information related to the content of the user's operation (S12). In response to this, the content image control unit 72 starts matching the captured image with the registered data (S14), and starts acquiring the user's characteristics and interests based on the matching result (S16).

Here, the registered data includes three-dimensional data of the spatial structure of the event site, template images of actual objects including contestants, signboards, and products. A user's characteristics and interests include the user's position, seat, team that the user supports, signboards and products that the user pays attention to, and the like. The characteristics and interests of the user may be narrowed down by adding the user's personal information stored in the user information storage unit 100 to the matching result of the photographed image. As described above, the collation of the captured images may be performed at any time, such as at a predetermined timing or at a timing when a predetermined user operation is performed.

Then, the content image control unit 72 starts selecting content images suitable for the user's characteristics and interests (S18). Content images may also be selected at any time in response to acquisition of user characteristics and interests. However, instead of determining specific content images each time, the content images may be narrowed down step by step according to the acquired characteristics and interests.

For example, when the content image control unit 72 identifies the team that the user is cheering on based on the user's seat, first, the content image control unit 72 extracts a set of event content for the team as selection candidates. Then, when the user performs a zoom-in operation or the contestant enters a score, etc., and when the timing for determining a specific content image arrives, the content image control unit 72 selects a specific content image, such as an enlarged image of the player, from among the selection candidates. determine the appropriate content image.

The content image control unit 72 waits until the timing for selecting and displaying a specific content image arrives (N of S20), and when that timing comes (Y of S20), it generates the content image (S22 ), and transmits the data to the portable terminal 12 (S24). Note that in the example of FIG. 10, the flow is such that image generation and transmission are performed only once at the display timing of the content image, but the gist of the present embodiment is not limited to this.

For example, generation and data transmission may be continuously repeated for one content image. In this case, those processes may be terminated by further determining the timing to hide the content image. Also, although illustration is omitted, when a purchase operation is performed on a GUI for purchasing a product displayed on the portable terminal 12 as a content image, the event management server 10 may execute the trading process as described above. .

After that, the content image control unit 72 repeats the process of selecting a content image at an appropriate timing, generating data, and transmitting the data (N of S26, S20 to S24). When it becomes necessary to stop the processing related to the content image, such as when the user terminates the application, the event management server 10 terminates the processing (Y of S26).

According to the present embodiment described above, the event management server controls the content image to be displayed superimposed on the real-space real-time video displayed on the terminal carried by the user who has come to the event. Here, the event management server collects images captured by the mobile terminal and estimates the characteristics and interests of the user based on the images, thereby optimizing the content images to be displayed. As a result, the user can enjoy various presentations along with the event and easily use the presented information without being bothered by unnecessary content images.

In addition, by comparing the three-dimensional data of the spatial structure of the event site with the photographed image, the position and seat of the user are specified. By using the three-dimensional data that is originally necessary when expressing content images with AR, affinity with AR technology is increased, and realization with a low introduction barrier is possible.

In addition, even if the user does not clearly instruct the menu or the image, a shooting operation such as zooming in is performed, the relationship between the position and orientation of the user and the contestants, the situation of the event, the environment, etc. to select and display an appropriate content image. As a result, the user can access desired information and images without missing an event, and can increase opportunities to use services via the mobile terminal.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the above embodiments are merely examples, and that various modifications can be made to combinations of each component and each treatment process, and that such modifications are within the scope of the present invention. be.

1 event management system 10 event management server 12 portable terminal 14 virtual store management server 16 actual store management server 23 CPU 26 main memory 32 communication section 50 CPU 52 main memory 54 display section 60 communication 62 motion sensor 64 camera 66 position acquisition unit 70 terminal information acquisition unit 72 content image control unit 74 trading processing unit 76 information storage unit 78 contestant status information acquisition unit 80 event status information acquisition unit , 82 image data transmission unit, 90 verification unit, 92 image selection unit, 94 image generation unit, 110 operation reception unit, 111 photographing unit, 112 terminal information transmission unit, 114 status information acquisition unit, 116 image data reception unit, 118 display Section 120 Personal Information Storage Section 122 Image Data Storage Section.

Claims (19)

a terminal information acquisition unit that acquires data of an image captured by a terminal held by a user who has visited the venue of the event;
a content image control unit that acquires characteristics of a user by matching the registered data with the photographed image, and selects a content image to be displayed on the terminal according to the characteristics;
an image data transmission unit configured to transmit data of the selected content image to the terminal for display;
An event management server system comprising: further comprising a spatial information storage unit for storing three-dimensional data of the spatial structure of the event venue;
2. The event management server system according to claim 1, wherein the content image control unit acquires the seat category of the user by matching the captured image with the spatial structure. 3. The event management server system according to claim 2, wherein the content image control unit identifies a team supported by the user among the teams playing the event as the event based on the seat category. . The content image control unit generates the content image as a virtual object arranged in the spatial structure,
The image data transmission unit includes the position information of the virtual object with respect to the three-dimensional data of the spatial structure in the data of the content image and transmits the corresponding position in the captured image being displayed on the terminal. 4. The event management server system according to claim 2, wherein an image of said virtual object is superimposed and displayed. the spatial information storage unit stores data of the spatial structure in a plurality of patterns;
5. The event management server system according to any one of claims 2 to 4, wherein the content image control unit switches the data of the spatial structure used for the matching according to a predetermined rule. The terminal information acquisition unit further acquires information on a user operation for changing the zoom magnification of the captured image,
6. The event management server system according to any one of claims 1 to 5, wherein the content image control unit selects the content image when a user operation to change the zoom magnification is performed. The terminal information acquisition unit further acquires information of a user operation indicating one of the captured images being displayed on the terminal,
7. The event management server system according to claim 1, wherein said content image control section selects said content image corresponding to the designated image. 8. The event management according to any one of claims 1 to 7, wherein the content image control unit identifies a subject that the user is paying attention to by the collation, and selects the content image corresponding to the subject. server system. Further comprising a contestant state information acquisition unit that acquires state information including the positions of contestants at the event venue,
9. The event management server system according to any one of claims 1 to 8, wherein the content image control section changes the content image according to the positional relationship between the user and the contestants. 10. The content image control unit, as the content image, selects an enlarged image representing the state of the contestant, and changes an enlargement ratio of the enlarged image according to the positional relationship. The described event management server system. 10. The content image control unit selects, as the content image, an image representing the appearance of the contestant viewed from a virtual viewpoint different from the user's viewpoint, which is determined based on the positional relationship. 11. The event management server system according to 10. further comprising an event status information acquisition unit that acquires the status of the event;
12. The event management server system according to any one of claims 1 to 11, wherein said content image control section changes the content of said content image according to the situation of said event. 12. The content image control unit selects an image of a discount coupon for a related product as the content image, and changes the discount rate indicated by the discount coupon according to the status of the event. The event management server system described in . 14. The event management server system according to claim 12, wherein the content image control unit selects an image representing a virtual effect corresponding to the situation of the event as the content image. 15. The event management server system according to any one of claims 1 to 14, wherein said content image control unit selects, as said content image, an image representing information related to a product related to a subject appearing in said photographed image. . 16. The event management server system according to claim 15, wherein the content image control unit determines the related product by detecting a signboard or product corresponding to the registration data from the photographed image. The content image control unit selects a content image for accepting purchase of the related product,
The terminal information acquisition unit further acquires information on the purchase operation of the related product performed at the terminal,
The event management server system is
a user information storage unit that stores correspondence information of the personal information of the terminal and the user;
a trading processing unit that realizes a trading procedure by acquiring the personal information associated with the terminal when the purchase operation is performed on the terminal;
17. The event management server system according to claim 15 or 16, further comprising: a step of acquiring data of an image captured by a terminal held by a user who has visited the venue of the event;
a step of acquiring characteristics of a user by matching the registered data with the photographed image, and selecting a content image to be displayed on the terminal according to the characteristics;
a step of transmitting data of the selected content image to the terminal for display;
A content image control method by an event management server system, comprising: A function to acquire image data captured by a terminal held by a user who has visited the venue of the event;
a function of acquiring characteristics of a user by matching registered data with the captured image, and selecting a content image to be displayed on the terminal according to the characteristics;
a function of transmitting data of the selected content image to the terminal for display;
A computer program characterized by causing a computer to implement

Images