JP6864593B2 - Information processing device - Google Patents

Description

The present invention relates to a technique for superimposing and displaying an image in a real space.

So-called AR (Augmented Reality) expands the real world seen by humans by adding, deleting, emphasizing, and attenuating information to captured images in real space. For example, in Patent Document 1, a spherical icon is superimposed on the captured image to notify the user of the existence of point information, and when the icon is selected by the user, the captured image is viewed from inside the sphere. It is disclosed to change the image.

Japanese Unexamined Patent Publication No. 2014-6582

An object of the present invention is to visually inform a user of a place worth paying attention to in a real space in an easy-to-understand manner.

In order to solve the above problems, the present invention presents a display control unit that superimposes and displays a floating body in the sky above each area included in the real space, and the height of the floating body that is superposed and displayed on the real space. Information processing comprising an altitude determination unit that determines according to information about the area corresponding to the above, and the altitude determination unit determines the altitude of the floating body according to the volume of sound in the area. Provide the device.

Further, the present invention has a display control unit that superimposes and displays a floating body in the sky above each area included in the real space, and the area corresponding to the floating body by displaying the altitude of the floating body superposed on the real space. Information processing including an altitude determination unit that determines the altitude according to information about the floating body, wherein the altitude determination unit determines the altitude of the floating body according to the remaining time of an event held in the area. Provide the device.

Further, the present invention has a display control unit that superimposes and displays a floating body in the sky above each area included in the real space, and the area corresponding to the floating body by displaying the altitude of the floating body superposed on the real space. The altitude determination unit is provided with an altitude determination unit that determines the altitude of the floating body according to the possibility of participating in an event held in the area. Provides an information processing device.

Further, the present invention has a display control unit that superimposes and displays a floating body in the sky above each area included in the real space, and the area corresponding to the floating body with the altitude of the floating body superimposed and displayed in the real space. The altitude determination unit includes an altitude determination unit that determines the altitude according to information about the floating body, and the altitude determination unit determines the altitude of the floating body according to the number of image pickup devices that image the real space including the area. To provide an information processing device.

According to the present invention, it is possible to visually and easily inform a place in the real space that deserves the attention of the user.

It is a figure which illustrates the structure of the information processing system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the hardware composition of the image pickup display device. It is a block diagram which shows an example of the hardware composition of an information processing apparatus. It is a figure which shows an example of the area DB stored in the information processing apparatus. It is a block diagram which shows an example of the functional structure of an information processing system. It is a sequence chart which shows an example of the operation of an information processing system. It is a figure which illustrates the real space imaged by the image pickup display device. It is a figure which illustrates the state that the floating body is superposed and displayed in the real space imaged by the image pickup display device.

[Constitution]
FIG. 1 is a diagram showing an example of the information processing system 1 of the present embodiment. The information processing system 1 includes an information processing device 10 that performs processing for realizing so-called AR (augmented reality), an imaging display device 20 that captures and displays an image, and a person who grasps the number of people in each area. The trend grasping equipment 30 is provided with a network 90 for communicably connecting the information processing device 10, the imaging display device 20, and the human trend grasping equipment 30. The network 90 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or a combination thereof, and includes a wired section or a wireless section. Although FIG. 1 shows one information processing device 10, one imaging display device 20, and one human trend grasping device 30, each of these may be plural.

The information processing device 10 controls the image pickup display device 20 so as to acquire an image captured in the real space captured by the image pickup display device 20 and display the image in a state in which an additional image is superimposed on the real space. Specifically, the information processing device 10 superimposes and displays a floating body floating at an altitude corresponding to information about the area above each area included in the real space. The floating body may have a shape such as a sphere or a polyhedron, or may have any shape such as a shape imitating a balloon, a balloon, an airship, or the like. The altitude of the floating body is determined according to the information about each area, but in the present embodiment, it is determined according to the number of people in each area grasped by the human trend grasping equipment 30. The greater the number of humans in an area, the higher the altitude of the floating body that is superimposed and displayed so as to float above the area. When the user who uses the image pickup display device 20 holds the image pickup display device 20 over the real space, the more people are gathered in the area, the more conspicuous the floating body is displayed at a higher position in the sky. Therefore, it is possible to visually understand which area is worth paying attention to.

FIG. 2 is a diagram illustrating a hardware configuration of the image pickup display device 20. The image pickup display device 20 is, for example, a glass-type, head-mounted display-type or contact-lens-type wearable computer, or a portable computer such as a smartphone or tablet. Devices that realize AR include a type that superimposes an additional image called an AR object on the display surface that displays the captured image in the real space and displays it, and a light transmission that is placed in front of the user's eyes. On the display surface of the mold, there is a type in which the AR object is displayed at a position on the display surface that overlaps with a predetermined position in the real space that can be seen from the user through the display surface. May be good. In any case, the image pickup display device 20 functions as a device for superimposing and displaying the AR object in the real space.

The image pickup display device 20 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an auxiliary storage device 204, a communication IF 205, an image pickup unit 206, a display unit 207, and an operation unit 208. It has a positioning unit 209 and an orientation detection unit 210.

The CPU 201 is a processor that performs various calculations. The ROM 202 is, for example, a non-volatile memory that stores programs and data used for starting the image pickup display device 20. The RAM 203 is a volatile memory that functions as a work area when the CPU 201 executes a program. The auxiliary storage device 204 is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores programs and data used in the image pickup display device 20. When the CPU 201 executes this program, the image pickup display device 20 functions as a computer device, and the functions shown in FIG. 5 described later are realized. The communication IF 205 is an interface for performing communication via the network 90 according to a predetermined communication standard. This communication standard may be a wireless communication standard or a wired communication standard. In the case of wireless, the communication IF205 includes an antenna and a communication circuit that operate according to a communication standard such as LTE (Long Term Evolution) or Wi-Fi (registered trademark). The image pickup unit 206 includes a lens and an optical element, and generates captured image data indicating the captured image. The display unit 207 includes, for example, a liquid crystal panel and a liquid crystal drive circuit, and displays an image according to the image data. The operation unit 208 includes an operator such as a key or a touch sensor, receives an operation of the user, and supplies a signal corresponding to the operation to the CPU 201. The positioning unit 209 is, for example, a GPS (Global Positioning System) unit, and measures the position of the imaging display device 20. The orientation detection unit 210 is, for example, a gyro sensor or an acceleration sensor, and detects the posture (orientation) of the image pickup display device 20.

FIG. 3 is a diagram illustrating a hardware configuration of the information processing device 10. The information processing device 10 is a computer device having a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, and a communication IF 105. The CPU 101 is a processor that performs various calculations. The ROM 102 is, for example, a non-volatile memory that stores programs and data used for starting the information processing apparatus 10. The RAM 103 is a volatile memory that functions as a work area when the CPU 101 executes a program. The auxiliary storage device 104 is a non-volatile storage device such as an HDD or SSD, and stores programs and data used in the information processing device 10. When the CPU 101 executes this program, the function shown in FIG. 5 described later is realized. The communication IF 105 is an interface for performing communication via the network 90 according to a predetermined communication standard. The information processing device 10 may include other configurations such as a display unit and an operation unit in addition to the configurations illustrated in FIG.

The auxiliary storage device 104 includes an area ID for identifying each area partitioned by a predetermined standard, the number of humans in the area, and the altitude of the floating body determined according to the number of humans. The area database (hereinafter, the database is abbreviated as DB) is stored. FIG. 4 is a diagram showing an example of the area DB. For example, in the area with the area ID "A01", there are "2500 people", and the altitude of the floating body determined according to the number is "100 m". The number of humans in each area is notified from the human trend grasping equipment 30 to the information processing device 10 and stored in the auxiliary storage device 104. The human trend grasping device 30 estimates the number of humans in each area from, for example, the area information of the mobile phone carried by each human. In this estimation, concealment processing such as removing or modifying ID information or the like for identifying the user is performed so that the user of the mobile phone cannot be identified.

FIG. 5 is a block diagram showing an example of the functional configuration of the information processing system 1. As shown in FIG. 5, in the image pickup display device 20, the functions of the image pickup unit 21 that images the real space and the display unit 22 that displays the image according to the image data under the control of the information processing device 10 are realized. To. In the information processing device 10, the acquisition unit 11 that acquires the number of humans in each area from the human trend grasping equipment 30, and the altitude of the floating body floating in the sky of the corresponding area are determined based on the acquired number. A function of an altitude determination unit 12 to be processed and a display control unit for displaying a floating body superimposed on each area included in the real space in the image pickup display device 20 is realized.
The determination unit determines the altitude of the floating body according to the number of humans in the area or the number or amount of actions by humans in the area.

In the image pickup display device 20, the image pickup unit 21 is realized by the image pickup unit 206 as hardware and the communication IF 205, and the display unit 22 is realized by the display unit 207 as hardware. In the information processing device 10, the acquisition unit 11 is realized by the communication IF 105 as hardware, the altitude determination unit 12 is realized by the CPU 101 as hardware, and the display control unit 13 is realized by the CPU 101 as hardware and the communication IF 105. To.

[motion]
An operation example of the information processing system 1 will be described with reference to FIGS. 6 to 8. First, the user activates the imaging unit 21 of the imaging display device 20 and starts imaging the real space around the user (step S11). At this time, the positioning unit 209 and the orientation detection unit 210 perform positioning and orientation detection, respectively (step S12). The imaging unit 21 transmits the captured image data obtained by imaging the real space to the information processing device 10 together with the information indicating the positioned position and the detected direction (step S13).

Here, FIG. 7 is a diagram illustrating an image captured by the image pickup display device 20. The captured image includes a plurality of buildings B and a road R.

When the display control unit 13 of the information processing device 10 acquires the captured image data or the like, it recognizes the position and orientation of the image pickup display device 20 and identifies the range imaged by the image pickup display device 20 (step S14). Then, the display control unit 13 compares the range imaged by the image pickup display device 20 with the area DB, and identifies the area included in the range. Further, the display control unit 13 identifies the altitude of the floating body corresponding to the specified area, transmits information including the position and altitude of the area corresponding to the floating body to the imaging display device 20, and in that area, the display control unit 13 transmits the information including the position and altitude of the floating body. The imaging display device 20 is instructed to display the floating body at an altitude (steps S15 and S16).

The display unit 22 of the image pickup display device 20 displays a floating body at a designated altitude in a designated area under the control of the display control unit 13 of the information processing device 10 (step S17). For example, when the imaging display device 20 is a type that displays an AR object at a position on a display surface that overlaps a predetermined position in the real space that can be seen from the user through a transparent display surface, the altitude specified in each area is high. The floating object is displayed at a position on the display surface that overlaps the position.

Here, FIG. 8 is a diagram illustrating a state in which a floating body is superimposed and displayed in the real space imaged by the image pickup display device 20. In the captured image, a plurality of buildings B and roads R, as well as floating bodies V1 to V4 are displayed. Of these floating bodies, the floating body V1 has the highest altitude, so that a large number of humans are gathering on the ground below the floating body V1. The user who sees this relies on the position of the floating body to go to the area by himself or search for information about the area.

According to the embodiment described above, the larger the number of human beings in the area, the higher the altitude of the floating body that is superimposed and displayed so as to float above the area. You can visually understand which area is worth noting.

[Modification example]
The present invention is not limited to the embodiments described above. The above-described embodiment may be modified as follows. Further, the following two or more modified examples may be combined and carried out.
[Modification 1]
The altitude determination unit 12 may determine the altitude of the floating body according to conditions other than the number of humans in each area. For example, the altitude determination unit 12 may determine the altitude of the floating body according to the number or amount of actions by humans in each area. The number or amount of actions by humans in each area referred to here is, for example, the number or amount of articles or comments posted by a person in that area on SNS (Social Networking Service) or the like. Since the function of recording the posting position when posting to SNS or the like is well known, the altitude determination unit 12 should use the function to specify the altitude according to the number or amount of posts by humans in each area. Just do it. In this way, an area with a large number or amount of human activity can be used as a noteworthy place.

Further, the altitude determination unit 12 may determine the altitude of the floating body according to the volume of the sound in each area. Specifically, the volume of voice in each area is detected by a microphone arranged in each area (for example, a microphone of a mobile phone carried by a human being may be used), and the altitude determination unit 12 uses the detection result. Then, the altitude of the floating body may be specified. For example, the louder the volume, the higher the altitude of the floating body. In this way, a loud area can be used as a noteworthy place.

Further, the altitude determination unit 12 may determine the altitude of the floating body according to the remaining time of the event held in each area. Specifically, the holding time and the ending time of the event held in each area may be described in the area DB, and the altitude determination unit 12 may specify the altitude of the floating body by using the description contents. .. For example, the longer the remaining time of the event, the higher the altitude of the floating body. In this way, an area with a long remaining time of the event can be used as a noteworthy place.

In addition, the altitude determination unit 12 may determine the altitude of the floating body according to the possibility of participating in an event held in each area. Specifically, the holding time and the end time of the event held in each area are described in the area DB, and the altitude determination unit 12 captures the position where the event is held, the holding time, and the end time. Compare with the position of the user using the display device 20, calculate the time that can participate in the event from the position of the user toward the area of the event, and use that time to identify the altitude of the floating body. do it. For example, the longer you can participate in an event, the higher the altitude of the floating body. In this way, an area that is likely to attend the event can be used as a noteworthy place.

Further, the altitude determination unit 12 may determine the altitude of the floating body according to the number of image pickup devices that image the real space including each area. Specifically, the altitude determination unit 12 identifies the position of the imaging range of the imaging display device 20, and when each area is included in the imaging range, counts the area as the number of images captured. , The altitude of the floating body may be specified by using the counted number. The number of images of the area may be counted, for example, at predetermined intervals of 1 minute. Thereby, the number of image pickup devices that image the real space including each area is specified. For example, the higher the number of image pickup display devices 20 that image the real space including each area, the higher the altitude of the floating body. In this way, as a place that deserves attention, an area with a large number of image pickup devices that image the real space including each area can be used.

Further, the altitude determination unit 12 may determine the altitude of the floating body according to the money paid as advertising expenses for the goods or services provided in each area. Specifically, the advertising cost of the product or service provided in each area may be described in the area DB, and the altitude determination unit 12 may specify the altitude of the floating body by using the advertising cost. For example, the higher the advertising cost, the higher the altitude of the floating body. In this way, the area where the advertising costs are paid a lot can be used as a place that deserves attention.

[Modification 2]
The display control unit 13 bursts and causes the floating body to burst, for example, when the user grips the floating body that is superimposed and displayed in the real space (that is, the act of breaking a balloon). A moving image in which a plurality of small floating bodies are floating from the floating bodies may be superimposed and displayed in the real space. Further, when the user touches these small floating bodies, the display control unit 13 provides information about the area (for example, the position of the area or the route toward the area) from the floating body. Alternatively, the name of the event being held in that area, etc.) may be superimposed and displayed.

Further, when the floating body is selected by an act such that the floating body superimposed and displayed in the real space is touched by the user, the information processing device 10 provides information on the area corresponding to the floating body (for example, the area). A processing unit that performs processing for sharing a position, a route to the area, an event name held in the area, etc.) among a plurality of users may be provided. For example, the processing unit performs a process of posting the above information to the SNS or the like used by the user.

Further, when the floating body is selected by an act such that the floating body superimposed and displayed in the real space is touched by the user, the information processing device 10 provides the user with value information regarding the area corresponding to the floating body. A processing unit that performs processing for providing may be provided. For example, the processing unit transmits information about the area (for example, the position of the area, the route toward the area, the name of the event held in the area, etc.) to the image pickup display device 20 by e-mail.

[Modification 3]
The size, color and shape of the floating body are arbitrary. Further, the size, color and shape of the floating body may be changed according to some conditions.

[Other variants]
The block diagram used in the description of the above embodiment shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. (For example, wired and / or wireless) may be connected and realized by these a plurality of devices. For example, at least a part of the functions of the information processing device 10 may be mounted on the image pickup display device 20. When all the functions of the information processing device 10 are mounted on the image pickup display device 20, the image pickup display device 20 functions as the information processing device according to the present invention.

Each aspect / embodiment described herein includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA. (Registered Trademarks), GSM®, CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), It may be applied to Bluetooth®, other systems that utilize suitable systems and / or next-generation systems that are extended based on them.

The order of the processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in the present specification may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order, and are not limited to the particular order presented.
Each aspect / embodiment described in the present specification may be used alone, in combination, or switched with execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.

The terms "system" and "network" as used herein are used interchangeably.

The information or parameters described herein may be represented by absolute values, relative values from a predetermined value, or other corresponding information. For example, the radio resource may be indexed.

The names used for the above parameters are not limited in any way. Further, mathematical formulas and the like using these parameters may differ from those expressly disclosed herein. Since the various channels (eg, PUCCH, PDCCH, etc.) and information elements (eg, TPC, etc.) can be identified by any suitable name, the various names assigned to these various channels and information elements are in any respect. However, it is not limited.

The terms "determining" and "determining" as used herein may include a wide variety of actions. “Judgment” and “decision” are, for example, judgment (judging), calculation (calculating), calculation (computing), processing (processing), derivation (deriving), investigating (investigating), and searching (looking up) (for example, a table). , Searching in a database or another data structure), ascertaining can be regarded as "judgment" or "decision". In addition, "judgment" and "decision" are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. (Accessing) (for example, accessing data in memory) may be regarded as "determination" or "decision". In addition, "judgment" and "decision" mean that "resolving", "selecting", "choosing", "establishing", "comparing", etc. are regarded as "judgment" and "decision". Can include. That is, "judgment" and "decision" may include that some action is regarded as "judgment" and "decision".

The present invention may be provided as an information processing method including steps of processing performed in the information processing system 1. Further, the present invention may be provided as a program executed by the image pickup display device 20 or the information processing device 10. Such a program may be provided in the form of being recorded on a recording medium such as an optical disk, or may be provided in the form of being downloaded to a computer via a network such as the Internet and being installed and made available. It is possible.

Software, instructions, etc. may be transmitted and received via a transmission medium. For example, the software uses wired technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described herein may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

The terms described herein and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings. For example, the channel and / or symbol may be a signal. Also, the signal may be a message. Further, the component carrier (CC) may be referred to as a carrier frequency, a cell, or the like.

Any reference to elements using designations such as "first", "second" as used herein does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Therefore, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

The "means" in the configuration of each of the above devices may be replaced with a "part", a "circuit", a "device" and the like.

As long as "inclusion," "comprising," and variations thereof are used herein or in the claims, these terms are as comprehensive as the term "comprising." Intended to be targeted. Furthermore, the term "or" as used herein or in the claims is intended not to be an exclusive OR.

Throughout this disclosure, if articles are added by translation, for example a, an, and the in English, unless the context clearly indicates that these articles are not. It shall include more than one.

Although the present invention has been described in detail above, it is clear to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as an amendment and a determination mode without departing from the spirit and scope of the present invention determined by the description of the claims. Therefore, the description of the present specification is for the purpose of exemplification and does not have any limiting meaning to the present invention.

1 ... Information processing system, 10 ... Information processing device, 11 ... Acquisition unit, 12 ... Altitude determination unit, 13 ... Display control unit, 20 ... Imaging display device, 21 ... Imaging unit, 22 ... Display unit, 30 ... Human trend grasping Equipment 90 ... Network, 101 ... CPU, 102 ... RAM, 103 ... ROM, 104 ... Auxiliary storage device, 105 ... Communication IF, 201 ... CPU, 202 ... RAM, 203 ... ROM, 204 ... Auxiliary storage device, 205 ... Communication IF , 206 ... Imaging unit, 207 ... Display unit, 208 ... Operation unit, 209 ... Positioning unit, 210 ... Direction detection unit.

Claims (4)

A display control unit that superimposes and displays floating objects in the sky above each area included in the real space,
It is provided with an altitude determination unit that determines the altitude of the floating body superimposed and displayed in the real space according to the information about the area corresponding to the floating body .
The altitude determination unit is an information processing device that determines the altitude of the floating body according to the volume of voice in the area. A display control unit that superimposes and displays floating objects in the sky above each area included in the real space,
With an altitude determination unit that determines the altitude of the floating body superimposed and displayed in the real space according to the information about the area corresponding to the floating body.
With
Before SL altitude determination unit, depending on the remaining time of the event is being held in the area, information processing apparatus characterized by determining the altitude of the floating body. A display control unit that superimposes and displays floating objects in the sky above each area included in the real space,
With an altitude determination unit that determines the altitude of the floating body superimposed and displayed in the real space according to the information about the area corresponding to the floating body.
With
Before SL altitude determination unit, depending on the possibilities that may participate in an event that has been held in the area, information processing apparatus you characterized by determining the altitude of the floating body. A display control unit that superimposes and displays floating objects in the sky above each area included in the real space,
With an altitude determination unit that determines the altitude of the floating body superimposed and displayed in the real space according to the information about the area corresponding to the floating body.
With
Before SL altitude determination unit sets a physical space including the areas according to the number of imaging devices that are captured, information processing apparatus characterized by determining the altitude of the floating body.

Images