JP6615732B2 - Information processing apparatus and image generation method - Google Patents

Description

  The present invention relates to a technique for outputting image data of an application to a display device such as a head-mounted display or a technique for generating image data to be provided to a display device.

  Patent Document 1 discloses a system in which a user and another user share a three-dimensional virtual reality space. In this system, the client terminal is provided with data on basic objects constituting the virtual reality space from the information server terminal, and is provided with data on the other person's avatar from the shared server terminal. The virtual reality space visible from the position is displayed on the display device. In Patent Document 1, a basic object is an object that is used in common by a plurality of users in a virtual reality space, for example, an object whose basic state does not change, such as a building or a road.

US Pat. No. 6,057,856

  By constructing a system in which a plurality of users can participate in the virtual reality space, the service providing entity can provide various experiences to the plurality of users. For example, if a classroom where a plurality of users participate is constructed as a virtual reality space, the user can take classes with other users while at home. In addition, if overseas landmarks and townscapes are constructed as a virtual reality space, multiple users can enjoy traveling abroad together.

  In a virtual reality space (hereinafter also referred to as “virtual space”), each user can freely move each character. For this reason, even if the service providing entity sets a target object (object) to be noted in the virtual space, the positions and line-of-sight directions of each character are different. It was difficult to get attention.

  This invention is made | formed in view of such a subject, The objective is to provide the technique for suggesting the direction which the user character should pay attention in virtual space.

  In order to solve the above problems, an information processing apparatus according to an aspect of the present invention is an information processing apparatus that outputs image data of an application to a display device, and acquires operation information for moving a user character in a virtual space And an image providing unit that outputs image data including a character of a user different from the user to the display device. The image providing unit outputs image data including a direction image indicating a direction in which another user's character is viewing and / or a direction in which the character is viewed to the display device.

  Another aspect of the present invention is an image generation method. This method includes a step of obtaining operation information for moving a user character in a virtual space, a step of obtaining operation information for moving a user character different from the user in the virtual space, and the user character. And a step of controlling the movement of the user's character different from the user based on the respective operation information, and based on the position of the user's character in the virtual space, Generating image data including a direction image indicating a direction in which another user's character is viewing and / or a direction in which the character is viewed.

  Note that any combination of the above components, the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded so as to be readable, a data structure, etc. are also included in the present invention It is effective as an embodiment of

  ADVANTAGE OF THE INVENTION According to this invention, the technique for suggesting the direction which the user character should pay attention in virtual space can be provided.

It is a figure which shows the information processing system in an Example. It is a figure which shows an example of the image of the virtual space seen from the viewpoint of the user character. It is a figure which shows the structural example of the user system in an Example. It is a figure which shows the example of the external appearance shape of HMD. It is a figure which shows the functional block of HMD. It is a figure which shows the functional block of information processing apparatus. It is a figure which shows an example of the image of the virtual space seen from the viewpoint of the user character. It is a figure which shows the example of a screen which included the direction image in the image which rendered three-dimensional virtual space. It is a figure which shows another example of the screen which included the direction image in the image which rendered three-dimensional virtual space.

  FIG. 1 shows an information processing system 1 in the embodiment. The information processing system 1 provides a service in which a plurality of users share a three-dimensional virtual space. The information processing system 1 includes a plurality of user systems 3a, 3b,... 3m, 3n (hereinafter, collectively referred to as “user system 3”) and a server system 4. The server system 4 is operated and managed by a service providing entity, and each user system 3 is used by each user. The server system 4 and the plurality of user systems 3 are connected to a network 2 such as the Internet, and the server system 4 and each user system 3 can transmit / receive data to / from each other. A plurality of user systems 3 may be able to transmit / receive data to / from each other.

  The server system 4 includes a management server 5 and a storage device 6. The storage device 6 includes a content DB 7 that holds 3D image data for generating a 3D virtual space, and a user behavior DB 8 for storing user behavior information in the 3D virtual space. Hereinafter, the three-dimensional image data stored in the content DB 7 is referred to as “content data”.

  For example, the server system 4 provides a three-dimensional virtual space of a classroom to the user system 3, the operator of the server system 4 becomes a teacher, and the users of the plurality of user systems 3 become students, so that a plurality of users can take classes together. We can provide services that can be received. In addition, the server system 4 provides the user system 3 with a three-dimensional virtual space of a city or a sightseeing spot, and an operator becomes a tour guide and a plurality of users become travel participants, so that a plurality of users can enjoy traveling with a guide. Service can be provided. Further, the server system 4 provides the user system 3 with a three-dimensional virtual space in which treasure is hidden, and an operator becomes a treasure hunting guide, and a plurality of users become treasure hunters. A service to compete for treasure hunting can be provided.

  As described above, the server system 4 provides a service for sharing a three-dimensional virtual space among a plurality of users. The content data stored in the content DB 7 is basically composed of static object data whose position information and posture information are not updated. For example, the content data representing the virtual space of a sightseeing spot includes a building object, a road object, a streetlight object, a signboard object, and the like whose relative positional relationships are defined. There may be an object that the user can move in the virtual space.

  The user system 3 installs a viewer application for reproducing the content data before receiving the service. When the user system 3 downloads content data from the server system 4 via the network 2, the viewer application constructs a virtual space based on the content data, and moves a character in the virtual space based on user operation information. The virtual space is rendered according to the position and posture (line-of-sight direction) of the user character. As a result, the user can freely move in the virtual space through the user character and view an image from a free viewpoint.

  FIG. 2 shows an example of a virtual space image viewed from the viewpoint of a certain user character. The viewer application generates an image from the viewpoint of the user character according to the position and line-of-sight direction of the user character.

  Each user system 3 downloads content data, thereby realizing sharing of a virtual space by a plurality of users. Hereinafter, an example in which ten users A, B, C, D, E, F, G, H, I, and J share a virtual space will be described. The management server 5 is in charge of virtual space sharing processing by a plurality of users. Each user operates each user character in the virtual space, and the operation information is used for processing by the viewer application in the user system 3 of the user and is transmitted to the management server 5. And transfer to the user system 3 of another user. That is, the operation information that the user A has operated his / her character is also provided to the user systems 3 of the other nine users B to J, and is used for processing by the viewer application of each user system 3. Thus, by sharing the operation information of the characters by each user to all users, the user characters of all users A to J in each user system 3 basically exhibit the same behavior except for the influence of communication delay. It becomes like this.

  FIG. 3 shows a configuration example of the user system 3 in the embodiment. The user system 3 includes an information processing device 10, a head mounted display device (HMD) 100, an input device 16 operated by a user's finger, an imaging device 14 that captures a user wearing the HMD 100, and an output device that displays an image. 15. The information processing apparatus 10 is connected to the network 2 via an access point (AP) 17. The AP 17 has functions of a wireless access point and a router, and the information processing apparatus 10 may be connected to the AP 17 with a cable or may be connected with a known wireless communication protocol.

  The HMD 100 is worn on the user's head and provides the user with a virtual reality (VR) world. By providing the HMD 100 with a head tracking function and updating the display screen in conjunction with the movement of the user's head, a sense of immersion in the video world can be enhanced. In the embodiment, the rendered image is displayed on the display panel of the HMD 100, but the rendered image may be displayed on the output device 15.

  The information processing apparatus 10 includes a processing device 11, an output control device 12, and a storage device 13. The processing device 11 is a terminal device that receives operation information input to the input device 16 by a user and executes a viewer application. The processing device 11 and the input device 16 may be connected by a cable or may be connected by a known wireless communication protocol. Note that the processing device 11 according to the embodiment also has a function of accepting the attitude information of the HMD 100 as user operation information for the viewer application and executing the viewer application. The output control device 12 is a processing unit that outputs the image data generated by the processing device 11 to the HMD 100. The output control device 12 and the HMD 100 may be connected by a cable or may be connected by a known wireless communication protocol.

  The imaging device 14 is a stereo camera, captures a user wearing the HMD 100 at a predetermined cycle, and supplies the captured image to the processing device 11. As will be described later, the HMD 100 is provided with a marker (tracking LED) for tracking the user's head, and the processing device 11 detects the movement of the HMD 100 based on the position of the marker included in the captured image. The HMD 100 is equipped with posture sensors (acceleration sensor and gyro sensor), and the processing device 11 acquires sensor data detected by the posture sensor from the HMD 100, so that it can be used with high accuracy together with the use of the captured image of the marker. Execute tracking process. Various methods have been proposed for tracking processing, and the processing device 11 may adopt any tracking method as long as it can detect the movement of the HMD 100.

  In the user system 3 of the embodiment, a viewer application that provides the HMD 100 with an image viewed from the viewpoint of the user character is executed. In the embodiment, since the user views the image with the HMD 100, the output device 15 is not necessarily required in the user system 3. However, in the modified example, the user does not wear the HMD 100, and the output control device 12 or the processing device 11 outputs the image. You may make it output from the apparatus 15.

  The HMD 100 is a display device that displays an image on a display panel positioned in front of the user when the user wears the head. The HMD 100 separately displays a left-eye image on the left-eye display panel and a right-eye image on the right-eye display panel. These images constitute parallax images viewed from the left and right viewpoints, and realize stereoscopic viewing. Since the user views the display panel through the optical lens, the information processing apparatus 10 supplies the HMD 100 with parallax image data in which optical distortion caused by the lens is corrected. Either the processing device 11 or the output control device 12 may perform this optical distortion correction processing.

  The processing device 11, the storage device 13, the output device 15, the input device 16, and the imaging device 14 may construct a conventional game system. In this case, the processing device 11 is a game device that executes a game, and the input device 16 is a device that supplies operation information by the user to the processing device 11 such as a game controller, a keyboard, a mouse, and a joystick. By adding the output control device 12 and the HMD 100 to the components of the game system, the user system 3 that implements the VR viewer application is constructed.

  Note that the function of the output control device 12 may be incorporated into the processing device 11 as a part of the function of the viewer application. That is, the processing unit of the information processing apparatus 10 may be configured by one processing apparatus 11 or may be configured by the processing apparatus 11 and the output control apparatus 12. In the following, functions of the processing device 11 and the output control device 12 necessary for implementing the viewer application will be collectively described as functions of the information processing device 10.

  As a stage before executing the viewer application, the processing device 11 downloads the content data stored in the storage device 6 of the server system 4 to the storage device 13. In addition, the user sets his / her character when starting the viewer application, and the set user character data is stored in the storage device 13 and transmitted to the server system 4 from the server system 4. It is transmitted to the user system 3.

  The viewer application moves the user character in the virtual space based on the operation information input from the input device 16, and determines the line-of-sight direction of the user character based on the posture information of the HMD 100 mounted on the user's head. In the embodiment, the viewer application uses the posture information of the HMD 100 as operation information for operating the gaze direction of the user character, but uses the input information in the input device 16 as operation information for operating the gaze direction. May be. Further, the operation information for moving the user character may use the position information of the HMD 100 without using the input information of the input device 16. In the embodiment, the operation information of the user character may be acquired from either the input device 16 or the HMD 100, or may be acquired from another user interface.

  The information processing apparatus 10 detects the position coordinates and orientation of the user's head (actually the HMD 100) by performing the user's head tracking process. Here, the position coordinates of the HMD 100 are position coordinates in a three-dimensional space with the reference position as the origin, and the reference position may be a position coordinate (latitude, longitude) when the power of the HMD 100 is turned on. Further, the attitude of the HMD 100 is an inclination in the three-axis direction with respect to a reference attitude in a three-dimensional space. The reference posture is a posture in which the user's line-of-sight direction is the horizontal direction, and the reference posture may be set when the power of the HMD 100 is turned on.

  The information processing apparatus 10 can detect the position coordinates and orientation of the HMD 100 only from the sensor data detected by the orientation sensor of the HMD 100, and further, by analyzing the image of the marker (tracking LED) of the HMD 100 captured by the imaging device 14, The position coordinates and orientation of the HMD 100 can be detected with high accuracy. As described above, the information processing apparatus 10 may calculate the position of the user character in the virtual space based on the position information of the HMD 100, and may calculate the gaze direction of the user character based on the posture information of the HMD 100. In the embodiment, the position of the user character in the virtual space is calculated based on the operation information of the input device 16.

  FIG. 4 shows an example of the external shape of the HMD 100. The HMD 100 includes an output mechanism unit 102 and a mounting mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that goes around the head when the user wears and fixes the HMD 100 to the head. The wearing band 106 has a material or a structure whose length can be adjusted according to the user's head circumference.

  The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 100, and includes a display panel that faces the eyes when worn. The display panel may be a liquid crystal panel or an organic EL panel. The housing 108 is further provided with a pair of left and right optical lenses that are positioned between the display panel and the user's eyes and expand the viewing angle of the user. The HMD 100 may further include a speaker or an earphone at a position corresponding to the user's ear.

  On the outer surface of the housing 108, light emitting markers 110a, 110b, 110c, and 110d are provided. In this example, the tracking LED constitutes the light emitting marker 110, but it may be any other type of marker, and in any case, the image is taken by the imaging device 14 and the information processing device 10 can perform image analysis of the marker position. I just need it. The number and arrangement of the light emitting markers 110 are not particularly limited, but need to be the number and arrangement for detecting the posture of the HMD 100. In the illustrated example, the light emitting markers 110 are provided at the four corners on the front surface of the housing 108. Further, the light emitting marker 110 may be provided on the side or the rear of the wearing band 106 so that the user can take a picture even when the user turns his back to the imaging device 14.

  The HMD 100 may be connected to the information processing apparatus 10 with a cable or may be connected with a known wireless communication protocol. The HMD 100 transmits the sensor data detected by the posture sensor to the information processing apparatus 10, receives image data generated by the information processing apparatus 10, and displays the image data on the left-eye display panel and the right-eye display panel.

  FIG. 5 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, audio data, sensor data, and commands. The storage unit 122 temporarily stores data, commands, and the like that are processed by the control unit 120. The posture sensor 124 detects posture information of the HMD 100. The posture sensor 124 includes at least a triaxial acceleration sensor and a triaxial gyro sensor.

  The communication control unit 128 transmits data output from the control unit 120 to the external information processing apparatus 10 by wired or wireless communication via a network adapter or an antenna. In addition, the communication control unit 128 receives data from the information processing apparatus 10 through wired or wireless communication via a network adapter or an antenna, and outputs the data to the control unit 120.

  When the control unit 120 receives image data and audio data from the information processing apparatus 10, the control unit 120 supplies the image data and audio data to the display panel 130 for display, and supplies them to the audio output unit 132 for audio output. The display panel 130 includes a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images is displayed on each display panel. Further, the control unit 120 causes the communication control unit 128 to transmit the sensor data from the attitude sensor 124 and the audio data from the microphone 126 to the information processing apparatus 10.

  FIG. 6 shows functional blocks of the information processing apparatus 10. The information processing apparatus 10 includes a sensor data receiving unit 20, a captured image receiving unit 22, an input data receiving unit 24, a server data receiving unit 26, an operation information transmitting unit 30, an image providing unit 32, and an audio as an input / output interface with the outside. A providing unit 34 is provided. The server data receiving unit 26 is a communication interface that receives data transmitted from the management server 5.

  The information processing apparatus 10 further includes an acquisition unit 40, a processing unit 50, and a storage device 13. The acquisition unit 40 has a function of acquiring information used in the viewer application from data received by the input / output interface, and includes a first operation information acquisition unit 42, a second operation information acquisition unit 44, and an object information acquisition unit 46. The processing unit 50 has a function of generating image data from the information acquired by the acquisition unit 40, and includes a character control unit 52, a determination unit 54, and an image generation unit 56. The storage device 13 stores content data for constructing a three-dimensional virtual space, character data of a user of the information processing device 10, character data of other users, and the like.

  In FIG. 6, each element described as a functional block for performing various processes can be configured by a circuit block, a memory, and other LSIs in terms of hardware, and loaded in the memory in terms of software. Realized by programs. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  Hereinafter, the operation of the information processing apparatus 10 used by “user A” will be described. The first operation information acquisition unit 42 acquires operation information for moving the user A's character in the virtual space. On the other hand, the second operation information acquisition unit 44 is operation information for moving a character other than the user A in the virtual space, that is, the characters of the users B, C, D, E, F, G, H, I, and J. To get.

  In the information processing apparatus 10 used by “user B”, the first operation information acquisition unit 42 acquires operation information for moving the character of user B, and the second operation information acquisition unit 44 Acquires operation information for moving another user's character.

Hereinafter, the operation information by the user A will be described.
The sensor data receiving unit 20 receives sensor data from the attitude sensor 124 of the HMD 100 worn by the user A at a predetermined cycle, and supplies the sensor data to the first operation information acquisition unit 42. The captured image receiving unit 22 receives an image obtained by capturing the HMD 100 at a predetermined cycle from the imaging device 14 and supplies the image to the first operation information acquisition unit 42. For example, the imaging device 14 captures a front space every (1/60) second, and the captured image receiving unit 22 receives a captured image every (1/60) second. The input data receiving unit 24 receives key data input by the user A from the input device 16 and supplies the key data to the first operation information acquisition unit 42.

  The first operation information acquisition unit 42 detects a change in the posture of the HMD 100 based on the movement of the HMD 100 mounted on the user's head, and acquires operation information indicating a change in the line-of-sight direction of the user A character in the virtual space. To do. Further, the first operation information acquisition unit 42 acquires operation information indicating the movement amount and movement direction of the user A character in the virtual space based on the key data of the input device 16 received by the input data receiving unit 24.

  The first operation information acquisition unit 42 detects a change in the attitude of the HMD 100 from the sensor data of the attitude sensor 124. The first operation information acquisition unit 42 may specify the posture change of the HMD 100 from the sensor data of the three-axis gyro sensor. Note that the first operation information acquisition unit 42 preferably further increases the detection accuracy of the posture change by further using the imaging result of the tracking light emitting marker 110. The first operation information acquisition unit 42 acquires operation information indicating a change in the viewing direction of the user A from the detected change in the attitude of the HMD 100.

  The first operation information acquisition unit 42 acquires operation information indicating the movement amount and movement direction of the character of the user A from the key data input to the input device 16. A known technique may be used for the process of moving the character in the virtual space by operating the input device 16. In the embodiment, since the user A wears the HMD 100, the line-of-sight direction of the character is matched with the line-of-sight direction of the user A wearing the HMD 100. However, in the modification, an embodiment in which the user A does not wear the HMD 100 is also assumed. In this case, the user A may input key data for controlling the position and line-of-sight direction of the character from the input device 16.

  In this way, the first operation information acquisition unit 42 acquires operation information for moving the user A's character in the virtual space. This operation information may be information that specifies the amount of change in the position and the line-of-sight direction, but may be information that directly specifies the position and the line-of-sight direction. The first operation information acquisition unit 42 supplies operation information for moving the user A's character to the character control unit 52. What kind of operation information the first operation information acquisition unit 42 acquires depends on the viewer application, and any operation information that can be processed by the character control unit 52 may be used. The first operation information acquisition unit 42 may generate operation information at a predetermined cycle and supply the operation information to the character control unit 52. However, when the state change occurs, that is, when the attitude of the HMD 100 changes or the input device 16 Operation information may be generated and supplied to the character control unit 52 when a key input occurs.

  Based on the operation information supplied from the first operation information acquisition unit 42, the character control unit 52 controls the movement of the character of the user A in the virtual space, specifically the position and line-of-sight direction of the character.

  The image generation unit 56 performs three-dimensional rendering of the virtual space based on the position (three-dimensional virtual space coordinate value) of the user character in the virtual space. Specifically, the image generation unit 56 generates image data by arranging a virtual camera having the optical axis in the direction of the line of sight of the user character at the position of the user character. The image providing unit 32 outputs the generated image data to the HMD 100.

  In the viewer application of the embodiment, the user A can change the line-of-sight direction of the character by moving the character by key input of the input device 16 and tilting the neck to change the direction of the HMD 100. The display panel 130 of the HMD 100 displays an image corresponding to the character position and line-of-sight direction (see FIG. 2).

  The above is the motion control of the character of the user A, but the information processing system 1 of the embodiment provides a service for sharing the three-dimensional virtual space among a plurality of users. Therefore, the operation information of the user A character acquired by the first operation information acquisition unit 42 is transmitted from the operation information transmission unit 30 to the management server 5, and is transmitted from the management server 5 to the information processing apparatuses 10 of other users B to J. Transferred.

  Also in the information processing apparatuses 10 of the other users B to J, operation information for moving each user's character is acquired and transmitted to the management server 5. The management server 5 transmits the operation information of the character acquired by the information processing apparatuses 10 of the users B to J to the information processing apparatus 10 of the user A. The server data receiving unit 26 receives character operation information of another user transmitted from the management server 5.

  The management server 5 manages information on users who participate in the virtual space sharing service. The user information includes at least a user account for identifying the user and character data of the user, and the management server 5 transmits the user information of each participating user to the information processing apparatuses 10 of all the participating users before starting the service. Keep it. After the service is started, the management server 5 associates the operation information of the user character with the user account and transmits it to each information processing apparatus 10. Therefore, the information processing apparatus 10 determines which operation information is received by the server data receiving unit 26. It is possible to specify whether the information is to be reflected in the movement of the user's character.

  The second operation information acquisition unit 44 acquires operation information for moving the characters of the other users B to J from the server data reception unit 26 and supplies the operation information to the character control unit 52. Based on the operation information supplied from the second operation information acquisition unit 44, the character control unit 52 controls the movement of another user's character in the virtual space, specifically the position and line-of-sight direction of the character. The control of the character's line-of-sight direction is performed by controlling the direction of the character's neck.

  As described above, the image generation unit 56 generates an image of the virtual space that is visible in the line-of-sight direction of the character from the position of the character of the user A. Therefore, if another user's character is included in the visual field of the user A's character, the image generation unit 56 generates image data including a user's character different from the user A. The image providing unit 32 outputs the generated image data to the HMD 100 that is a display device.

  FIG. 7 shows an example of an image of the virtual space viewed from the viewpoint of the user A character. Here, a state where five user characters are included in the field of view of the user A character is shown. Each user character is assumed to be a character of user B, user D, user F, user G, and user H in order from the left. For the other users C, E, I, and J, the viewing angle of the user A's character is because it is behind the user A's character or in the building. It is not included in the image data. In the HMD 100, the communication control unit 128 receives image data from the information processing apparatus 10, and the control unit 120 displays a virtual space image on the display panel 130.

  In the embodiment, the server system 4 provides a virtual space of a sightseeing spot to a plurality of user systems 3, an operator of the server system 4 becomes a tour guide, and a plurality of users become travel participants so that a plurality of users can guide. Provide a service to travel together. The operator operates the management server 5 to distribute a voice guide to each user system 3 and designates the direction in which the user moves and the target (object) to be noticed by voice. In each user system 3, when the server data receiving unit 26 receives voice data for voice guidance, the voice providing unit 34 transmits the voice data to the HMD 100. In HMD 100, communication control unit 128 receives voice data, and control unit 120 outputs voice guidance from voice output unit 132 to an earphone (not shown).

The operator performs the following voice guidance, for example.
“Go straight ahead in the arcade that follows.”
“The buildings on both sides of the passage were built in the late 17th century.”
“The cobblestones at the feet remain as they were at the time.”
Each user operates the character according to the voice guidance, and the character moves in the virtual space. On the HMD 100 of each user, the virtual space image is changed and displayed as the character moves.

  In addition, since movement of a character is based on operation of each user, the position in the virtual space of each user character is various. The management server 5 receives character operation information from all users, and the operator knows the positions of the characters of the respective users. Therefore, the operator can individually provide voice guidance to a user who has deviated from a predetermined moving route, delayed or moved too far.

  When there is an object to be noted in such a virtual space, an operator who is a tour guide performs voice guidance so as to pay attention to the object. It may be "Please pay attention to the signboard beside the aisle". An “explanation event” is started by this voice guidance. In the screen example shown in FIG. 7, the object 200 is set as an object to be noticed, and when the operator confirms that all users are looking at the object 200, the point of interest of the object 200 (for example, a signboard store) Explaining history). When the description of the point of interest ends, the explanation event ends, and movement guidance to the next target object is performed. The start and end of the explanation event may be performed at an arbitrary timing by the operator.

  When the voice guidance “Please pay attention to the signboard beside the passage” is performed, if the character is in the vicinity of the object 200, the user can find the object 200 relatively easily. When it is away from the user, it is difficult for the user to find the object 200 immediately. Therefore, the viewer application includes an image for suggesting the direction of attention of the user character in the display image so that the user can easily recognize the position of the object 200.

  When the operator of the management server 5 transmits the voice data “Please pay attention to the signboard beside the passage” to the information processing apparatuses 10 of a plurality of users, the identifier of the object 200 (object ID) and the object 200 The three-dimensional coordinates in the virtual space are also transmitted. Hereinafter, the object ID and the object three-dimensional coordinates are referred to as “object information”.

  In the information processing apparatus 10, when the server data receiving unit 26 receives the audio data and the object information, the audio providing unit 34 transmits the audio data to the HMD 100, the object information acquiring unit 46 acquires the object information, and the determining unit 54 To supply. The audio data transmitted to the HMD 100 is output by the audio output unit 132.

  The determination unit 54 determines whether another user is currently looking at the object 200 based on the position and line-of-sight direction of another user's character. This determination process is a process for determining whether or not the object 200 is shot by the virtual camera when a virtual camera having the optical axis in the direction of the line of sight of the character is placed at the position of another user's character. . Basically, the determination unit 54 determines that another user is looking at the object 200 if the three-dimensional coordinates of the object 200 are included in the viewing angle of the character, while the object 200 is within the viewing angle. Is not included, it is determined that another user is not looking at the object 200. The determination unit 54 hides the object 200 by another object (obstacle) between the character and the object 200 even when the three-dimensional coordinates of the object 200 are included in the viewing angle of the character. In this case, it is determined that another user is not looking at the object 200.

  With the above determination processing, the determination unit 54 can specify whether or not a user character different from the user A is looking at the object 200. In the screen example illustrated in FIG. 7, the determination unit 54 determines that the characters of the user B, the user D, the user F, and the user G are looking at the object 200, while the character of the user H is looking at the object 200. Judge that there is no. The determination unit 54 provides the determination result to the image generation unit 56.

  The determination process by the determination unit 54 may be performed periodically, and the determination cycle may be the same as the character's motion control cycle. That is, the character control unit 52 periodically controls the actions of the user A's character and another user's character, but the determination unit 54 may perform the determination process in accordance with the cycle.

  The image generation unit 56 generates image data including a direction image indicating the direction in which another user's character is looking based on the determination result by the determination unit 54. Here, when another user's character is looking at the object 200, the direction image may indicate the line-of-sight direction of the other user's character, but the direction from the other user's character toward the object 200 may be It may be shown. When another user's character is not looking at the object 200, the direction image is an image showing the line-of-sight direction of the character.

  The direction of the character's line of sight and the direction from the character toward the object 200 will be described. In the embodiment, the line-of-sight direction of the character is a direction in which the HMD 100 worn by the user is facing and a direction extending from the center of both eyes of the character. With regard to the screen displayed on the HMD 100, the line-of-sight direction of the character is a direction that cuts the center of the display screen vertically.

  On the other hand, the direction from the character toward the object 200 will be described. The display screen shown in FIG. 7 is a screen displayed on the HMD 100 of the user A, but the object 200 is included slightly in the upper left from the screen center. In this case, the direction from the character toward the object 200 is not parallel to the line-of-sight direction. When the user A moves the HMD 100 so that the object 200 is positioned at the center of the display screen, the line-of-sight direction coincides with the direction from the character toward the object 200.

  FIG. 8 shows an example of a screen in which a direction image is included in an image obtained by rendering a three-dimensional virtual space. The image generation unit 56 includes direction images 202b, 202d, 202f, and 202g that indicate directions from the respective characters of the user B, the user D, the user F, and the user G toward the object 200, and a direction that indicates the line-of-sight direction of the user H character. Image data including the image 202h is generated. Here, the direction images 202b, 202d, 202f, and 202g to be added to the character viewing the object 200 and the direction image 202h to be added to the character not viewing the object 200 are generated so as to have different display modes. Is done.

  In the screen example shown in FIG. 8, the direction images 202b, 202d, 202f, and 202g are solid line arrow images, and the direction image 202h is a dotted line arrow image, but the display mode is not limited to this. For example, the direction images 202b, 202d, 202f, and 202g may be expressed in different colors such as red and the direction image 202h in blue. In any case, by changing the display mode, the user A can easily recognize which character is looking at the object 200 and which character is not looking at the object 200.

  In the screen example shown in FIG. 8, the object 200 is already included in the display screen viewed by the user A, but the user A may not notice the object 200. By including the directional images 202b, 202d, 202f, and 202g on the display screen, the user A can recognize at a glance which object to focus on. Since the direction image 202h does not face the object 200, the image generation unit 56 may not include the direction image 202h in the image data.

  FIG. 9 shows another screen example in which a direction image is included in an image obtained by rendering a three-dimensional virtual space. Although the case where the signboard at the upper left of the screen is an attention object has been described with reference to FIG. 8, an example in which the signboard is not actually the attention object and another signboard is the attention object will be described.

  At this time, the characters of the user B, the user D, the user F, and the user G are paying attention to the wrong object. Therefore, the direction image added to each character is a dotted arrow image indicating that the object of interest is not viewed.

  Here, the direction image added to the character of the user H is a solid line arrow image, which indicates that the object of interest is correctly viewed. As a result, the user A who sees the screen shown in FIG. 9 recognizes that the object of interest exists beyond the line of sight of the user H.

  When it is desired to pay attention to an object in the virtual space, it is also effective to apply an effect that makes the object stand out (for example, blinks). However, if the object is not included in the display screen, the user cannot recognize the effect. Therefore, as shown in FIG. 9, by adding a direction image to the user's character who is correctly paying attention to the object, the user can check the location of the object not included in the display screen.

  Even if you take a tour in the real world, you may not know what to pay attention to because you didn't listen to the tour guide. In such a case, the person imagines what should be noted from the movements of those around him. In the embodiment, by adding a direction image to another user's character, the user A can easily recognize the object of interest, and the tour participant in the virtual space can find the object of interest with a sense similar to the real world. .

  In the embodiment, the direction image is represented by an arrow, but the present invention is not limited to this. For example, in order to express the viewing angle of the character, a direction image of a cone extending to the object of interest with the character position as a vertex may be used. The direction image may be given a translucent color to ensure the visibility of the spatial image serving as the background of the direction image.

  Note that once the user recognizes the object, the user does not always continue to see the object until the explanation from the operator is started. If the arrival of other users at the object installation point is delayed and the explanation of the operator has not started yet, it is also a pleasure of sightseeing to look around the surrounding scene. Therefore, when a certain user arrives at the object installation point with a delay, it is assumed that all other users do not see the object.

  Therefore, if the character has not seen the object of interest at the current time but has seen the object of interest in the past, the image generation unit 56 adds a direction image indicating the direction toward the object of interest to the character. Also good. For example, the image generation unit 56 may add a direction image indicating a direction toward the target object only within a predetermined time from the time when the character last viewed the target object. This direction image serves as a hint indicating the position of the object of interest, and assists the user in searching for an object that is delayed from the object installation position. However, since the user who arrives at the object installation position too late causes great trouble to other users, the display time of the direction image indicating the direction toward the object of interest is limited to the predetermined time, thereby limiting the assist. You may spend.

  Note that the image generation unit 56 may add a direction image indicating the direction toward the target object only when the character is within a predetermined distance from the position at the time when the target object was last viewed. If the character stays at the same position, a direction image may be added from that point. For example, when the character moves and walks around, the object of interest can no longer be seen. You may be in a position that is not. Therefore, the image generation unit 56 may determine whether or not to add a direction image indicating the direction toward the target object seen in the past according to the current position of the character.

  In addition, although the Example demonstrated the service in which a some user becomes a travel participant, the operator in the server system 4 or the person according to it may participate in a tour as a dummy participant. The dummy participant knows the whole of the trip and plays the role of assisting with a comfortable tour behind the scenes. For example, when no user can find the target object, the dummy participant looks at the target object, so that other users see the direction image from the dummy participant character toward the target object, The location of the object of interest can be recognized.

  It should be noted that instead of the operator actually participating in the tour as a dummy participant, the management server 5 may allow a character that operates according to a predetermined rule to participate in the tour. This character is a so-called non-player character (NPC) and is called a “non-operation character” in the embodiment. At this time, it is preferable that the service providing entity does not notify the user that the non-operation characters are mixed in with the participating characters.

  The management server 5 transmits operation information for moving the non-operation character to the information processing apparatus 10 of each user. In the information processing apparatus 10, when the server data receiving unit 26 receives the operation information of the non-operation character from the management server 5, the second operation information acquisition unit 44 acquires the operation information of the non-operation character, and the character control unit 52. To supply. The operation information of the non-operation character is handled in the same manner as the operation information of another user.

  When a predetermined condition is satisfied after the start of the explanation event, the management server 5 transmits operation information for moving the non-operation character so as to see the target object to each information processing apparatus 10. For example, if no user finds the target object within a predetermined time after the start of the explanatory event, the non-operation character operates so as to see the target object. As a result, a direction image indicating the direction from the non-operation character toward the target object is displayed, and the user can recognize the location of the target object. The use of the non-operation object is efficient because it is not necessary to operate the dummy participant character on the management server 5 side. In the management server 5, it is preferable that the non-operation character can be freely switched to the operation character by the dummy participant, or the operation character by the dummy participant can be freely switched to the non-operation character.

  In the above, this invention was demonstrated based on the Example. It is to be understood by those skilled in the art that the embodiments are exemplifications, and that various modifications can be made to combinations of the respective components and processing processes, and such modifications are within the scope of the present invention.

  In the embodiment, the direction in which the HMD 100 is directed is used as the user's line-of-sight direction. .

  In the service described in the embodiment, the input device 16 may be used so that the user can easily find the object of interest. For example, when a posture sensor is mounted on the input device 16 and the posture of the input device 16 is changed, the posture direction of the input device 16 in the actual three-dimensional space and the user character in the virtual three-dimensional space are directed to the target object. When the direction matches or approximates, the input device 16 may vibrate or a notification sound is output to notify the user of the direction of the object of interest. The advantage of using the input device 16 for object search is that the user wearing the HMD 100 does not have to move the neck to search for the object of interest. For example, when a direction image indicating a direction from another user character toward the target object is not displayed on the display panel 130 of the HMD 100, the target object can be quickly found by searching for the target object using the input device 16. become. For example, in a service such as a treasure hunt that aims to find an object, it is considered that such a function of the input device 16 is usefully used.

  Further, in the information processing system 1 of the embodiment, the processing function shown in FIG. In this case, the management server 5 becomes a cloud server having the function of the information processing apparatus 10 of the embodiment, and provides each user system 3 with individually generated image data.

DESCRIPTION OF SYMBOLS 1 ... Information processing system, 3 ... User system, 4 ... Server system, 5 ... Management server, 6 ... Storage device, 7 ... Content DB, 8 ... User behavior DB DESCRIPTION OF SYMBOLS 10 ... Information processing apparatus, 15 ... Output device, 20 ... Sensor data receiving part, 22 ... Captured image receiving part, 24 ... Input data receiving part, 26 ... Server data reception , 30... Operation information transmission unit, 32... Image providing unit, 34... Voice providing unit, 40... Acquisition unit, 42. 2 operation information acquisition unit, 46 ... object information acquisition unit, 50 ... processing unit, 52 ... character control unit, 54 ... determination unit, 56 ... image generation unit, 100 ... HMD .

Claims (6)

An information processing device that outputs image data of an application to a display device,
An acquisition unit for acquiring operation information for moving the user's character in the virtual space;
An image providing unit that outputs image data including a character of a user different from the user to a display device, and the image providing unit was viewing and / or viewing the character of another user the image data including the direction image indicating the direction, there is to be output to the display device,
In the virtual space, objects that the user should pay attention to are set,
If the object is included in the viewing angle of another user's character, the image data includes a direction image indicating the direction from the other user's character toward the object, and is within the viewing angle of the other user's character. If the object is not included in the image information, the image data includes a direction image indicating the direction of the line of sight of another user . The acquisition unit acquires operation information for moving another user's character in the virtual space,
The information processing apparatus
A control unit that controls the movement of the user's character in the virtual space, and the movement of the user's character different from the user based on the respective operation information acquired by the acquisition unit;
An image generation unit that generates image data including another user's character based on the position of the user's character in the virtual space;
The information processing apparatus according to claim 1. In the image data, a direction image added to a character watching and / or watching the object and a direction image added to a character not watching the object are displayed in different display modes. include,
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. A detection unit for detecting the posture of the head mounted display mounted on the user's head;
The acquisition unit acquires the posture information of the head mounted display detected by the detection unit as operation information for moving the user character.
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. Obtaining operation information for moving the user's character in the virtual space;
Obtaining operation information for moving a character of a user different from the user in the virtual space;
Controlling the movement of the user's character and the movement of a user's character different from the user based on each operation information;
Step of generating image data including another user's character and a direction image showing the direction of the other user's character and / or a direction of viewing based on the position of the user's character in the virtual space And having
The step of generating image data is as follows:
If a predetermined object is included within the viewing angle of another user character, the image data includes a direction image indicating the direction from the other user character toward the object, and the viewing angle of the other user character. If the object is not included in the image data, a direction image indicating the direction of the line of sight of another user is included in the image data . On the computer,
A function of acquiring operation information for moving the user's character in the virtual space;
A function of acquiring operation information for moving a character of a user different from the user in the virtual space;
A function for controlling the movement of the user's character and the movement of a user's character different from the user based on each operation information;
A function for generating image data including another user's character and a direction of the other user's character and / or a direction image indicating the direction of viewing based on the position of the user's character in the virtual space a program for causing the the realization,
The function to generate image data is
If a predetermined object is included within the viewing angle of another user character, the image data includes a direction image indicating the direction from the other user character toward the object, and the viewing angle of the other user character. If the object is not included in the program, the program includes a function of including, in the image data, a direction image indicating the direction of the line of sight of another user.

Images