JP6805375B2 - Information processing equipment, information processing methods and information processing programs - Google Patents

Description

Embodiments of the present invention relate to information processing devices, information processing methods and information processing programs.

There is an AR (Augmented Reality) application that superimposes and displays related information about an object provided in real space on an image. The AR application is installed and used in an information processing device. In this information processing device, it is desired that related information can be displayed in an easy-to-read manner.

Japanese Unexamined Patent Publication No. 2014-206904

An embodiment of the present invention provides an information processing device, an information processing method, and an information processing program capable of displaying information related to an object in an easy-to-see manner.

According to an embodiment of the present invention, an information processing device including an acquisition unit and a processing unit is provided. The acquisition unit acquires a first image at the first time and a second image at the second time after the first time. The first image includes a first image of the first object and a second image of the second object. The second image includes a third image of the first object and a fourth image different from the third image. The processing unit carries out the first operation and the second operation. The first operation generates first data in which the first related information associated with the first image is superimposed and displayed on the first image. The second operation is the relative first positional relationship of the second image with respect to the first image and the relative second position of the fourth image with respect to the third image. When the state of change between the positional relationship and the state of change is the first state, the second data for displaying the first related information superimposed on the second image is generated.

It is a block diagram which illustrates the information processing apparatus which concerns on 1st Embodiment. It is a schematic diagram which illustrates the relationship between an object and an image. 3A and 3B are schematic views showing an application example of the information processing apparatus according to the first embodiment. 4 (a) and 4 (b) are diagrams illustrating the operation of the detection unit 21 according to the first embodiment. 5 (a) and 5 (b) are diagrams illustrating the operation of the corresponding unit 22 according to the first embodiment. 6 (a) to 6 (d) are diagrams illustrating the operation of the calculation unit 23 according to the first embodiment. 7 (a) and 7 (b) are diagrams illustrating the operation of the display control unit 24 according to the first embodiment. 8 (a) to 8 (c) are schematic views illustrating the display result of the related information according to the first embodiment. It is a block diagram which illustrates the information processing apparatus which concerns on 2nd Embodiment. 10 (a) and 10 (b) are schematic views showing an application example of the information processing apparatus according to the second embodiment. 11 (a) and 11 (b) are diagrams illustrating the operation of the setting unit 25 according to the second embodiment. 12 (a) to 12 (c) are schematic views illustrating the display result of the related information according to the second embodiment. 13 (a) and 13 (b) are schematic views showing an application example of the information processing apparatus according to the third embodiment. 14 (a) and 14 (b) are diagrams illustrating the operation of the calculation unit 23 according to the third embodiment. 15 (a) and 15 (b) are diagrams illustrating the operation of the display control unit 24 according to the third embodiment. 16 (a) to 16 (c) are schematic views illustrating the display result of the related information according to the third embodiment. 17 (a) and 17 (b) are schematic views showing an application example of the information processing apparatus according to the fourth embodiment. 18 (a) and 18 (b) are diagrams illustrating the operation of the detection unit 21 according to the fourth embodiment. 19 (a) and 19 (b) are diagrams illustrating the operation of the corresponding unit 22 according to the fourth embodiment. It is a schematic diagram which illustrates the display result of the related information which concerns on 4th Embodiment. It is a block diagram which illustrates the information processing apparatus which concerns on 5th Embodiment. 22 (a) to 22 (c) are diagrams showing an application example of the information processing apparatus according to the fifth embodiment. 23 (a) to 23 (c) are schematic views illustrating the display results of the related information according to the fifth embodiment. It is a block diagram which illustrates the information processing apparatus which concerns on 6th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes between the parts, and the like are not necessarily the same as the actual ones. Further, even when the same parts are represented, the dimensions and ratios may be different from each other depending on the drawings.
In addition, in the present specification and each figure, the same elements as those described above with respect to the above-mentioned figures are designated by the same reference numerals, and detailed description thereof will be omitted as appropriate.

(First Embodiment)
FIG. 1 is a block diagram illustrating an information processing apparatus according to the first embodiment.
As shown in FIG. 1, the image processing device 110 according to the embodiment includes an acquisition unit 10, a processing unit 20, a display unit 30, a first storage unit 40, and a second storage unit 50. For example, an input / output terminal is used for the acquisition unit 10. The acquisition unit 10 includes an input / output interface that communicates with the outside via wired or wireless. For the processing unit 20, for example, an arithmetic unit including a CPU (Central Processing Unit), a memory, and the like is used. An integrated circuit such as an LSI (Large Scale Integration) or an IC (Integrated Circuit) chipset can be used for a part or all of each block of the processing unit 20. An individual circuit may be used for each block, or a circuit in which a part or the whole is integrated may be used. Each block may be provided integrally, or some blocks may be provided separately. In addition, a part of each block may be provided separately. For integration, not only LSI but also a dedicated circuit or a general-purpose processor may be used.

The processing unit 20 is provided with a detection unit 21, a corresponding unit 22, a calculation unit 23, and a display control unit 24. Each of these parts is realized as, for example, an image processing program. That is, the image processing device 110 can also be realized by using a general-purpose computer device as basic hardware. The functions of each part included in the image processing device 110 can be realized by causing the processor mounted on the computer device to execute the image processing program. At this time, the image processing device 110 may be realized by installing the above image processing program in the computer device in advance, storing the above image in a storage medium such as a CD-ROM, or via a network. It may be realized by distributing a processing program and appropriately installing this image processing program on a computer device. Further, the processing unit 20 is realized by appropriately using a memory, a hard disk or a storage medium such as a CD-R, a CD-RW, a DVD-RAM, or a DVD-R built in or externally to the above-mentioned computer device. Can be done.

In this example, the first storage unit 40 and the second storage unit 50 are represented separately, but the first storage unit 40 and the second storage unit 50 may be configured as one storage unit.

FIG. 2 is a schematic diagram illustrating the relationship between the object and the image.
As shown in FIG. 2, the image processing device 110 is integrally provided with an imaging unit 60 such as a digital still camera. The image pickup unit 60 may be provided separately from the image processing device 110. The imaging unit 60 images the first object 71 and the second object 72 provided in the real space RS to generate the first image 61 and the second image 62. The acquisition unit 10 acquires the first image 61 and the second image 62. The first image 61 and the second image 62 may be either a moving image or a still image.

The first image 61 is an image at the first time t1. The first image 61 includes a first image 81 of the first object 71 and a second image 82 of the second object 72. The second image 62 is an image at the second time t2 after the first time t1. The second image 62 includes a third image 83 of the first object 71 and a fourth image 84 different from the third image 83. In this example, both the first image 81 and the third image 83 are images of the first object 71. Both the second image 82 and the fourth image 84 are images of the second object 72.

In the present embodiment, the processing unit 20 implements the first operation Op1 and the second operation Op2. The first operation Op1 generates the first data d1 in which the first related information 41 associated with the first image 81 is superimposed and displayed on the first image 61. In the second operation Op2, when the state of change between the first positional relationship 51 and the second positional relationship 52 is the first state, the first related information 41 is superimposed and displayed on the second image 62. 2 Generate data d2. The first positional relationship 51 is a relative positional relationship of the second image 82 with respect to the first image 81. The second positional relationship 52 is a relative positional relationship of the fourth image 84 with respect to the third image 83. The first state includes, for example, a state in which there is no change or a small change between the first positional relationship 51 and the second positional relationship 52. That is, the first state includes a state in which the first positional relationship 51 and the second positional relationship 52 do not substantially change.

More specifically, the detection unit 21 carries out the first detection process. In the first detection process, the first image 81 and the second image 82 are detected from the first image 61. Correspondence unit 22 carries out association processing. In the associative processing, the first related information 41 regarding the first object 71 is associated with the first image 81. The calculation unit 23 carries out the first calculation process. In the first calculation process, the relative first positional relationship 51 of the second image 82 with respect to the first image 81 is calculated. The display control unit 24 executes the first display control process. In the first display control process, the first data d1 in which the first related information 41 is superimposed and displayed on the first image 62 is generated.

Further, the detection unit 21 carries out the second detection process. In the second detection process, the third image 83 and the fourth image 84 are detected from the second image 62. The calculation unit 23 executes the second calculation process. In the second calculation process, the relative second positional relationship 52 of the fourth image 84 with respect to the third image 83 is calculated. The display control unit 24 executes the second display control process. In the second display control process, when the state of change between the first positional relationship 51 and the second positional relationship 52 is the first state, the first related information 41 is superimposed and displayed on the second image 62. The second data d2 is generated. The display unit 30 displays the first image 61 and the second image 62.

As described above, according to the embodiment, in the first image 61, the target image (second image) around the target image (first image 81, hereinafter also referred to as a attention target image) for displaying related information is used as a reference. The relative positional relationship with 82) (first positional relationship 51) is obtained. Further, in the second image 62, the relative positional relationship (second positional relationship 52) with the target image (fourth image 84) around the third image 83 is obtained as a reference. If these positional relationships do not substantially change, the display of the related information (first related information 41) of the target image of interest is continued.

The image processing device 110 according to the embodiment is applied to AR application software in which an object existing in a real space is photographed by a camera or the like, and related information about the object is superimposed and displayed on the photographed image. The related information is a wide variety of information about the object, and includes, for example, work procedures for the object and incidental information that can be obtained via the Internet or the like.

Here, in the case of an image including an object image of many objects, if related information is superimposed and displayed on each of the object images, it becomes very difficult to see. On the other hand, there is a reference example in which the target image closest to the center of the angle of view (center of the image) of the camera or the portion corresponding to the line of sight of the user is set as the target image of interest, and related information is displayed only for the target image of interest.

However, in the above reference example, when displaying related information for a plurality of images continuously captured, the center of the angle of view of the camera or the line of sight of the user frequently occurs due to the movement of the user or the like. It can happen to move. For this reason, the image of interest changes from moment to moment, and the related information displayed is frequently switched, making it difficult to see.

On the other hand, according to the embodiment, for a plurality of images, the relative positional relationship between the target image of interest and the surrounding target image is obtained, and if these positional relationships do not substantially change, Information related to the image of interest is continuously displayed. Therefore, the related information displayed does not change frequently due to the movement of the user, and the related information can be displayed in an easy-to-read manner.

That is, as shown in FIGS. 1 and 2, the acquisition unit 10 acquires and acquires a plurality of images (for example, the first image 61 and the second image 62) of an object existing in the real space RS. A plurality of images are output to the detection unit 21. The acquisition unit 10 may acquire an image in real time from an image pickup unit 60 such as a digital still camera. The acquisition unit 10 may read and acquire image data recorded on a storage medium such as an HDD (Hard Disk Drive). For example, digital image data obtained by converting an image input from the imaging unit 60 is recorded in the storage medium.

The detection unit 21 analyzes the first image 61 input from the acquisition unit 10, detects the target image included in the first image 61, outputs the position of the target image to the calculation unit 23, and outputs the position of the target image to the calculation unit 23. The type is output to the corresponding unit 22 respectively. Here, the target image may include, for example, character information such as a signboard, a logo, a signboard, and a name plate. The target image may include, for example, shape information of an object such as a product placed in a store, a building standing in the city, or a machine installed in a factory. Further, the target image may include, for example, information about a person's face or a person.

The position of the target image output by the detection unit 21 is expressed as coordinate information on which the target image is located in the image. The coordinate information of the object at this time may be set so as to strictly surround the object image, or may be simplified by using an circumscribing rectangle, coordinates of the center of gravity, or the like. The type of the target image output by the detection unit 21 is information for distinguishing it from other target images. For example, the target image including the character information may be character-recognized, and the obtained character string may be used as the type of the target image. The object image including the shape information of the object may be shape-recognized, and the obtained object shape may be used as the type of the object image. Furthermore, the image portion itself cut out from the target image may be used as a type.

The corresponding unit 22 selects related information corresponding to each target image type from the related information 40a stored in the first storage unit 40, based on the type of each target image input from the detection unit 21. Then, the selected related information is output to the display control unit 24. For example, the first related information 41 is associated with the first image 81. The related information 40a is information such as characters corresponding to the object (object image). For example, in an AR application that translates Japanese characters into English characters, the English characters in the translation result correspond to related information. Further, in an AR application that displays a work procedure for a device, a work procedure manual explaining the operation procedure of the device, an image showing an outline of the operation, and the like correspond to related information. The related information 40a is not particularly limited as long as it can be superimposed and displayed on the image acquired in relation to the object.

The calculation unit 23 calculates the relative positional relationship 50a of the plurality of target images based on the positions of the plurality of target images input from the detection unit 21, and records the calculated positional relationship 50a in the second storage unit 50. To do. For example, the positional relationship 50a includes a relative first positional relationship 51 between the first image 81 and the second image 82 included in the first image 61. Here, the first image 81 is set as the target image of interest. The first positional relationship 51 is a relative positional relationship of the second image 82 with respect to the first image 81.

The display control unit 24 generates the first data d1 in which the first related information 41 associated with the first image 81 is superimposed and displayed on the first image 61. Then, the display control unit 24 controls the display unit 30 to display the first data d1 on the display unit 30.

The detection unit 21, the corresponding unit 22, and the calculation unit 23 repeat the same processing for the plurality of target images included in the second image 62, and the third image 83 and the fourth image 84 included in the second image 62. The relative second positional relationship 52 with and from is calculated.

The display control unit 24 compares the first positional relationship 51 with the second positional relationship 52, and determines whether or not the state of change in the relative positional relationship is the first state. In the first state, the first related information 41 is superimposed and displayed on the second image 62. At this time, the display position may be determined from the position of the target image as the starting point to which position the related information is to be displayed. On the other hand, if the second state is different from the first state, the first related information 41 is not displayed on the second image 62. Alternatively, the second image 62 may display another related information different from the first related information 41. As another related information, for example, the second related information regarding the second object 72 can be displayed. Here, the second state is a state in which the second positional relationship 52 is changed with respect to the first positional relationship 51.

In this way, it is possible to control whether the same related information is displayed continuously or switched to display another related information based on the relative positional relationship of a plurality of target images included in the image. Can be done. Therefore, even when a plurality of images are continuously input, the related information is not frequently switched, and the related information that is easy for the user to see can be displayed.

3A and 3B are schematic views showing an application example of the information processing apparatus according to the first embodiment.
FIG. 3A is a schematic diagram illustrating a plurality of objects.
FIG. 3B is a schematic view illustrating how the worker 80 moves from the point X to the point Z while taking an image of a plurality of objects with the information processing device 110.

As shown in FIG. 3A, there are a first object 71, a second object 72, and a third object 73 in the real space RS. As shown in FIG. 3B, the information processing device 110 is, for example, a tablet terminal provided with a camera (imaging unit 60). For example, the information processing device 110 displays the first related information 41 relating to the first object 71, and the worker 80 displays the first related information 41 on the first to third objects 71 to 73 while looking at the first related information 41. approach. The worker 80 moves in the order of point X, point Y, and point Z. The worker 80 is moving while holding the information processing device 110 over the first to third objects 71 to 73. The first image 61 (point X) includes the fifth image 85 of the third object 73 in addition to the first image 81 and the second image 82. The second image 62 (point Z) includes the sixth image 86 of the third object 73 in addition to the third image 83 and the fourth image 84.

In this example, the first operation Op1 generates the third data d3 in which the first related information 41 associated with the first image 81 is superimposed and displayed on the first image 61. In the second operation Op2, the state of change between the first positional relationship 51 and the second positional relationship 52 is the first state, and the relative first of the fifth image 85 when the first image 81 is used as a reference. When the state of change between the three positional relationships 53 and the relative fourth positional relationship 54 of the sixth image 86 with respect to the third image is the third state, the first related information 41 is provided. , The fourth data d4 to be superimposed and displayed on the second image 62 is generated. Similar to the first state, the third state includes, for example, a state in which there is no change or a small change between the third positional relationship 53 and the fourth positional relationship 54. That is, the third state includes a state in which the third positional relationship 53 and the fourth positional relationship 54 do not substantially change.

4 (a) and 4 (b) are diagrams illustrating the operation of the detection unit 21 according to the first embodiment.
FIG. 4A is a schematic diagram illustrating the first image 61.
FIG. 4B is a flowchart illustrating an operation example of the detection unit 21.

As shown in FIG. 4A, the information processing apparatus 110 images the first object 71, the second object 72, and the third object 73 at the point X, and images the first image 81 and the second image, respectively. The first image 61 including the 82 and the fifth image 85 is acquired and displayed on the screen. Here, the first object 71 is given a character label "Type B", the second object 72 is given a character label "Type A", and the third object 73 is given a character label "Type C". .. In this example, the character label is recognized as a character from the first image 61, the position of each target image (first image 81, second image 82, and fifth image 85) is calculated based on the character label, and each of them is calculated. Recognize the type of target image.

As shown in FIG. 4B, the detection unit 21 detects a plurality of target images (first image 81, second image 82, and fifth image 85) from the first image 61 (step S1). For example, character recognition is performed on the first image 61, and an image including characters is detected as an object image. The technique for detecting characters from an image is called scene character detection, and there are various methods. For example, the image is divided into a plurality of small areas, and a feature amount composed of a multidimensional vector is extracted for each of the divided small areas. As the feature amount, for example, a local feature amount such as SIFT (Scale-Invariant Feature Transform) can be used. For example, it is determined that the target image exists in a small area with a certain number of correspondences by associating a typical character image pattern (uppercase or lowercase alphanumeric characters) with the pattern in the registered database by the local feature amount. be able to. As a result, the first image 81 including "Type B", the second image 82 including "Type A", and the fifth image 85 including "Type C" are detected from the first image 61.

Next, the detection unit 21 calculates the position of each target image (first image 81, second image 82, and fifth image 85) (step S2). Here, the position of each target image is calculated based on a plurality of character groups. For example, a mass of adjacent characters is detected as a character string, and the center of gravity of the circumscribing rectangle containing the detected character string is calculated as the position of the target image.

Next, the detection unit 21 recognizes the type of each target image (first image 81, second image 82, and fifth image 85) (step S3). The type of each target image, that is, "Type B" of the first image 81, "Type A" of the second image 82, and "Type C" of the fifth image 85 is determined by the character pattern associated with the local feature amount. Recognize each.

5 (a) and 5 (b) are diagrams illustrating the operation of the corresponding unit 22 according to the first embodiment.
FIG. 5A is a diagram illustrating related information 40a.
FIG. 5B is a flowchart illustrating an operation example of the corresponding unit 22.

As shown in FIG. 5A, it is related to the type (“Type B”, “Type A”, “Type C”) of each target image (first image 81, second image 82, fifth image 85). A database for associating information 40a (first related information 41, second related information 42, third related information 43) is held. This database is stored in advance in the first storage unit 40. In this example, "Type B" of the first image 81 corresponds to "the first device to be inspected" (first related information 41) regarding the first object 71. "Type A" of the second image 82 corresponds to "Please confirm whether the meter is in the normal range" (second related information 42) regarding the second object 72. "Type C" of the fifth image 85 corresponds to "Please check if the switch is on" (third related information 43) regarding the third object 73.

As shown in FIG. 5B, the corresponding unit 22 searches the database (first storage unit 40) based on the type of each target image (step S11), and associates related information with each target image (step). S12). For example, the second image 82 detected from the first image 61 is given a type called "Type A". Therefore, the second related information 42, that is, "Please confirm whether the meter is in the normal range" is associated. Similarly, the first image 81 is given a type called "Type B". Therefore, the first related information 41, that is, "the device to be inspected first" is associated. The fifth image 85 is given a type called "Type C". Therefore, the third related information 43, that is, "Please check if the switch is on" is associated.

Here, the first image 81 is preset as a target image of interest for displaying related information. Therefore, the first related information 41 associated with the first image 81 is superimposed and displayed on the first image 61.

As shown in FIG. 3B, the same processing is performed on the second image 62 at the point Z. That is, the detection unit 21 detects a plurality of target images (third image 83, fourth image 84, and sixth image 86) from the second image 62. The detection unit 21 calculates the position of each target image and recognizes the type of each target image. The corresponding unit 22 searches the database (first storage unit 40) based on the type of each target image, and associates related information with each target image. For example, the fourth image 84 detected from the second image 62 is given a type called “Type A”. Therefore, the second related information 42, that is, "Please confirm whether the meter is in the normal range" is associated. Similarly, the third image 83 is given a type called "Type B". Therefore, the first related information 41, that is, "the device to be inspected first" is associated. The sixth image 86 is given a type called "Type C". Therefore, the third related information 43, that is, "Please check if the switch is on" is associated.

6 (a) to 6 (d) are diagrams illustrating the operation of the calculation unit 23 according to the first embodiment.
FIG. 6A is a flowchart illustrating an operation example of the calculation unit 23.
FIG. 6B is a schematic diagram illustrating the first relative coordinate system.
FIG. 6C is a schematic diagram illustrating a second relative coordinate system.
FIG. 6D is a diagram illustrating the calculation results of the first to fourth positional relationships.

As shown in FIG. 6A, the calculation unit 23 acquires the coordinates of each of the plurality of target images (first image 81, second image 82, fifth image 85) included in the first image 61 at the point X. (Step S21). Then, the calculation unit 23 determines the center point in the relative coordinate system in order to calculate the relative positional relationship of these plurality of target images (step S22). For example, as shown in FIG. 6B, a first relative coordinate system centered on the first position b1 in the first image 81 including the character string “Type B” closest to the center of the angle of view at the point X. Set c1. Then, the calculation unit 23 calculates the first positional relationship 51 and the third positional relationship 53 based on the first relative coordinate system c1 as shown in FIG. 6D (step S23).

Here, the first relative coordinate system c1 includes the first axis a1 and the second axis a2. The first axis a1 passes through the first position b1 and extends in the first direction D1. The second axis a2 extends in the second direction D2 through the first position b1 and intersecting the first direction D1. The first direction D1 is, for example, the X-axis direction of the first image 61. The second direction D2 is, for example, the Y-axis direction of the first image 61.

The first positional relationship 51 includes a first-direction hd1 and a second-direction hd2. The first direction hd1 is a direction along the first axis a1 in the third direction D3. The third direction D3 is a direction from the first image 81 to the second image 82. The second direction hd2 is a direction along the second axis a2 in the third direction D3. That is, if the second image 82 is on the right side of the second axis a2, the first orientation hd1 is to the right. The state in which the first direction hd1 is facing right is represented by "+". If the second image 82 is to the left of the second axis a2, the first orientation hd1 is to the left. The state in which the first direction hd1 is facing left is represented by "-". Further, if the second image 82 is above the first axis a1, the second orientation hd2 is upward. The state in which the second direction hd2 is upward is represented by "+". If the second image 82 is below the first axis a1, the second orientation hd2 is downward. The state in which the second direction hd2 is downward is represented by "-". In the case of the example of FIG. 6B, the positional relationship between the first image 81 (Type B) and the second image 82 (Type A) is (left side, upper side), so (first direction hd1, The second orientation hd2) is represented by (-, +).

The third positional relationship 53 includes a fifth orientation hd5 and a sixth orientation hd6. The fifth direction hd5 is a direction along the first axis a1 in the fifth direction D5. The fifth direction D5 is a direction from the first image 81 to the fifth image 85. The sixth direction hd6 is a direction along the second axis a2 in the fifth direction D5. That is, if the fifth image 85 is on the right side of the second axis a2, the fifth orientation hd5 is to the right. The state in which the fifth direction hd5 is facing right is represented by "+". If the fifth image 85 is on the left side of the second axis a2, the fifth orientation hd5 is to the left. The state in which the fifth direction hd5 is facing left is represented by "-". Further, if the fifth image 85 is above the first axis a1, the sixth orientation hd6 is upward. The state in which the sixth direction hd6 is upward is represented by "+". If the fifth image 85 is below the first axis a1, the sixth orientation hd6 is downward. The state in which the sixth direction hd6 is downward is represented by "-". In the case of the example of FIG. 6B, the positional relationship between the first image 81 (Type B) and the fifth image 85 (Type C) is (right side, upper side), so (fifth direction hd5, The sixth orientation hd6) is represented as (+, +).

The calculation unit 23 performs the same processing on the second image 62 at the point Z. That is, the calculation unit 23 acquires the coordinates of each of the plurality of target images (third image 83, fourth image 84, sixth image 86) included in the second image 62 (step S21). Then, the calculation unit 23 determines the center point in the relative coordinate system in order to calculate the relative positional relationship of these plurality of target images (step S22). For example, as shown in FIG. 6 (c), the second relative coordinate system c2 centered on the second position b2 of the third image 83 including the character string “Type B” closest to the center of the angle of view at the point Z. To set. Then, the calculation unit 23 calculates the second positional relationship 52 and the fourth positional relationship 54 based on the second relative coordinate system c2, as shown in FIG. 6D (step S23).

Here, the second relative coordinate system c2 includes the third axis a3 and the fourth axis a4. The third axis a3 passes through the second position b2 and extends in the first direction D1. The fourth axis a4 extends in the second direction D2 through the second position b2 and intersecting the first direction D2. The first direction D1 is, for example, the X-axis direction of the second image 62. The second direction D2 is, for example, the Y-axis direction of the second image 62.

The second positional relationship 52 includes a third orientation hd3 and a fourth orientation hd4. The third direction hd3 is a direction along the third axis a3 in the fourth direction D4. The fourth direction D4 is a direction from the third image 83 to the fourth image 84. The fourth direction hd4 is a direction along the fourth axis a4 in the fourth direction D4. That is, if the fourth image 84 is on the right side of the fourth axis a4, the third orientation hd3 is to the right. The state in which the third direction hd3 is facing right is represented by "+". If the fourth image 84 is on the left side of the fourth axis a4, the third orientation hd3 is to the left. The state in which the third direction hd3 is facing left is represented by "-". Further, if the fourth image 84 is above the third axis a3, the fourth orientation hd4 is upward. The state in which the fourth direction hd4 is upward is represented by "+". If the fourth image 84 is below the third axis a3, the fourth orientation hd4 is downward. The state in which the fourth direction hd4 is downward is represented by "-". In the case of the example of FIG. 6 (c), the positional relationship between the third image 83 (Type B) and the fourth image 84 (Type A) is (left side, upper side), so (third direction hd3, The fourth orientation hd4) is represented by (-, +).

The fourth positional relationship 54 includes a seventh orientation hd7 and an eighth orientation hd8. The seventh direction hd7 is a direction along the third axis a3 in the sixth direction D6. The sixth direction D6 is a direction from the third image 83 to the sixth image 86. The eighth direction hd8 is a direction along the fourth axis a4 in the sixth direction D6. That is, if the sixth image 86 is on the right side of the fourth axis a4, the seventh orientation hd7 is to the right. The state in which the 7th direction hd7 is facing right is represented by "+". If the sixth image 86 is on the left side of the fourth axis a4, the seventh orientation hd7 is leftward. The state in which the 7th direction hd7 is facing left is represented by "-". Further, if the sixth image 86 is above the third axis a3, the eighth orientation hd8 is upward. The state in which the eighth direction hd8 is upward is represented by "+". If the sixth image 86 is below the third axis a3, the eighth orientation hd8 is downward. The state in which the eighth direction hd8 is downward is represented by "-". In the case of the example of FIG. 6 (c), the positional relationship between the third image 83 (Type B) and the sixth image 86 (Type C) is (right side, upper side), so (7th direction hd7, The eighth direction hd8) is represented by (+, +).

7 (a) and 7 (b) are diagrams illustrating the operation of the display control unit 24 according to the first embodiment.
FIG. 7A is a flowchart illustrating an operation example of the display control unit 24.
FIG. 7B is a diagram illustrating the determination result of the target image of interest.

As shown in FIGS. 7 (a) and 7 (b), the display control unit 24 has a display control unit 24 between the first positional relationship 51 and the second positional relationship 52, and the third positional relationship 53 and the fourth positional relationship 54. It is determined whether or not there is a substantial change in each of the above (step S31), the related information to be displayed is determined based on the determination result (step S32), and the display position of the related information is determined. (Step S33).

In this example, the first direction hd1 and the third direction hd3 are the same, and the second direction hd2 and the fourth direction hd4 are the same. Therefore, the state of change between the first positional relationship 51 and the second positional relationship 52 is determined to be the first state. Further, the fifth direction hd5 and the seventh direction hd7 are the same, and the sixth direction hd6 and the eighth direction hd8 are the same. Therefore, the state of change between the third positional relationship 53 and the fourth positional relationship 54 is determined to be the third state. In this case, the display control unit 24 determines that there is no substantial change in the positional relationship at the point Z, and continuously displays the related information displayed at the point X. On the other hand, when it is determined that the positional relationship has changed at the point Z, the target image of interest is set again, and the related information to be displayed is switched to another related information.

At points X and Y, it is assumed that the target image of "Type B" has been determined as the target image of interest. When arriving at the point Z, it is determined whether or not the relative positional relationship has substantially changed. Comparing the positional relationship at the point X (and the point Y) with the positional relationship at the point Z, it can be seen that these positional relationships have not changed substantially. Therefore, it can be determined that the target image of interest is a “Type B” target image like the point X (and the point Y). In this case, the related information to be displayed can be determined as the first related information 41, that is, "the device to be inspected first".

The display control unit 24 may adaptively determine the display position of the related information. For example, a position around the target image of interest and not overlapping with other detected target images can be determined as a display position.

8 (a) to 8 (c) are schematic views illustrating the display result of the related information according to the first embodiment.
FIG. 8A is a diagram illustrating a display result of related information at the point X.
FIG. 8B is a diagram illustrating the display result of the related information at the point Y.
FIG. 8C is a diagram illustrating a display result of related information at the point Z.

As shown in FIG. 3A, the real space RS has a first object 71, a second object 72, and a third object 73. For example, the information processing device 110 displays the first related information 41 regarding the first object 71, and the worker 80 displays the first related information 41 on the first to third objects 71 to 73 while looking at the first related information 41. approach. The worker 80 moves in the order of point X, point Y, and point Z. The worker 80 is moving while holding the information processing device 110 over these plurality of objects.

As shown in FIG. 8A, the first image 61 (point X) includes the fifth image 85 of the third object 73 in addition to the first image 81 and the second image 82. As shown in FIG. 8C, the second image 62 (point Z) further includes the sixth image 86 of the third object 73 in addition to the third image 83 and the fourth image 84. There is no substantial change in the relative positional relationship between the plurality of object images at the points X and Z. Therefore, at the point Z, the first related information 41 displayed at the point X is continuously displayed.

In the above embodiment, it is assumed that the object is accurately detected in the first image of the point X, the second image of the point Y, and the third image of the point Z. However, even if an object is not detected for a moment in a continuous image, it is sufficient to determine whether or not the relative positional relationship has changed by complementing the undetected object with the images before and after. For example, when a plurality of buildings are imaged in a landscape seen by the user, the image may be blurred or the target building may be hidden for a moment due to an obstacle or the like because the user holding the imaging device moves. .. In this case, the detection objects of the preceding and following images may be complemented and dealt with, or all the captured images may not be processed and may be processed at predetermined frame intervals. Further, the same applies when an object moves, for example, a car or luggage moving on a predetermined course. For these, a means for switching according to a predetermined situation may be separately prepared, and processing may be added according to a user's instruction.

As described above, according to the embodiment, even if the positions of the plurality of target images that fit in the image change due to the movement of the worker, the same related information does not change as long as the relative positional relationship of the plurality of target images does not change. Can be displayed continuously. Therefore, it is possible to display the related information in an easy-to-read manner.

(Second Embodiment)
FIG. 9 is a block diagram illustrating an information processing apparatus according to the second embodiment.
As shown in FIG. 9, the information processing apparatus 111 according to the embodiment is further provided with a setting unit 25.

The processing unit 20 further performs an operation of deriving the first positional relationship 51. The derivation operation is performed when the distance between the center of the first image 81 and the center of the first image 61 is shorter than the distance between the center of the second image 82 and the center of the first image 61 for a predetermined time. , Including deriving the first positional relationship 51. The first positional relationship 51 is a relative positional relationship of the second image 82 with respect to the first image 81. The setting unit 25 sets the first image 81 as a reference. That is, the first image 81 is set as the target image of interest.

10 (a) and 10 (b) are schematic views showing an application example of the information processing apparatus according to the second embodiment.
FIG. 10A is a schematic diagram illustrating a plurality of objects.
FIG. 10B is a schematic view illustrating a state in which the worker 80 holds the information processing device 111 and images a plurality of objects.

As shown in FIG. 10A, in the real space RS, as a plurality of objects, a first object 71 including the character string “Type B” and a second object 72 including the character string “Type A” A third object 73 including the character string "Type C", a fourth object 74 including the character string "Type D", and a fifth object 75 including the character string "Type E" are provided. ing.

As shown in FIG. 10B, the information processing apparatus 111 displays the first image 61 in which the first to fifth objects 71 to 75 are captured. The first image 61 is the first image 81 of the first object 71, the second image 82 of the second object 72, the fifth image 85 of the third object 73, and the seventh image of the fourth object 74. Includes image 87 and eighth image 88 of the fifth object 75.

The information processing device 111 is, for example, a tablet terminal provided with a camera (imaging unit 60). As described above, it is assumed that a plurality of objects exist in the real space RS and the worker 80 holds the information processing device 111 over the plurality of objects. In the present embodiment, a setting unit 25 for setting an image of interest is added. An operation example of the setting unit 25 will be described below.

11 (a) and 11 (b) are diagrams illustrating the operation of the setting unit 25 according to the second embodiment.
FIG. 11A is a flowchart illustrating an operation example of the setting unit 25.
FIG. 11B is a schematic diagram illustrating the operation of the setting unit 25.

As shown in FIG. 11A, the setting unit 25 specifies the coordinates for estimating the position of the image of interest with respect to the first image 61 (step S41). For example, a touch panel or the like for inputting coordinates may be mounted on the display unit 30 for input. Alternatively, a specific area existing at the angle of view may be specified as the coordinates. Here, as shown in FIG. 11B, the center coordinates 61a of the angle of view are specified. Of the five images 81, 82, 85, 87, 88 corresponding to the first to fifth objects 71 to 75, the first image 81 of "Type B" is closest to the center coordinates 61a. Therefore, the first image 81 of "Type B" is determined as a candidate target image (step S42).

The setting unit 25 measures the duration (step S43). For example, when the threshold value is set to 2 seconds, the first image 81 is set as the target image of interest by fixing the angle of view of the information processing apparatus 111 for 2 seconds or longer (step S44).

That is, the distance between the center of the first image 81 and the center coordinate 61a of the first image 61 is shorter than the distance between the center of the second image 82 and the center coordinate 61a of the first image 61. When this positional relationship continues for a predetermined time, the first image 81 is set as the target image of interest. As a result, the first positional relationship 51 is derived. Regarding the positional relationship between the first image 81 and each of the fifth image 85, the seventh image 87, and the eighth image 88, the position between the first image 81 and the second image 82 is also obtained. Similar to a relationship.

Further, the display control unit 24 controls so that after the worker 80 explicitly sets the target image of interest, the display of the related information regarding the target image of interest is continued unless the relative positional relationship changes.

12 (a) to 12 (c) are schematic views illustrating the display result of the related information according to the second embodiment.
FIG. 12A is a diagram illustrating the display result of the related information at the point X.
FIG. 12B is a diagram illustrating the display result of the related information at the point Y.
FIG. 12C is a diagram illustrating the display result of the related information at the point Z.

As shown in FIG. 10A, there are 71st to 5th objects 71 to 75 in the real space RS. The points X, Y, and Z in this example are the same as the points X, Y, and Z shown in FIG. 3 (b). For example, the information processing device 111 displays the first related information 41 regarding the first object 71, and the worker 80 displays the first related information 41 on the first to fifth objects 71 to 75 while looking at the first related information 41. approach. The worker 80 moves in the order of point X, point Y, and point Z. The worker 80 is moving while holding the information processing device 111 over these plurality of objects.

As shown in FIG. 12A, the first image 61 (point X) includes the first image 81, the second image 82, the fifth image 85, and the seventh image 87 of the fourth object 74. Includes the eighth image 88 of the fifth object 75. As shown in FIG. 12 (c), the second image 62 (point Z) includes the third image 83, the fourth image 84, the sixth image 86, and the ninth image 89 of the fourth object 74. The tenth image 90 of the fifth object 75 is included. There is no substantial change in the relative positional relationship between the plurality of object images at the points X and Z. Therefore, at the point Z, the first related information 41 displayed at the point X is continuously displayed.

As described above, according to the embodiment, by providing the setting unit, the operator can set the desired target image of interest, and further, the display of the related information regarding the set target image of interest can be continued. It is possible to provide information in an easy-to-read manner.

(Third Embodiment)
13 (a) and 13 (b) are schematic views showing an application example of the information processing apparatus according to the third embodiment.
FIG. 13A is a schematic diagram illustrating a plurality of objects.
FIG. 13B is a schematic view illustrating how the worker 80 moves from the point X to the point Z while taking an image of a plurality of objects with the information processing device 112.

As shown in FIG. 13A, there are 71st to 5th objects 71 to 75 in the real space RS. As shown in FIG. 13B, the information processing device 112 is, for example, a tablet terminal provided with a camera (imaging unit 60). For example, the information processing apparatus 112 displays the first related information 41 regarding the first object 71, and the worker 80 displays the first related information 41 on the first to fifth objects 71 to 75 while looking at the first related information 41. approach. The worker 80 moves in the order of point X, point Y, and point Z. The worker 80 is moving while holding the information processing device 112 over these plurality of objects. In the first image 61 (point X), in addition to the first image 81, the second image 82, and the fifth image 85, the seventh image 87 of the fourth object 74 and the eighth image 88 of the fifth object 75 are obtained. including. In the second image 62 (point Z), in addition to the third image 83, the fourth image 84, and the sixth image 86, the ninth image 89 of the fourth object 74 and the tenth image 90 of the fifth object 75 including.

14 (a) and 14 (b) are diagrams illustrating the operation of the calculation unit 23 according to the third embodiment.
FIG. 14A is a flowchart illustrating an operation example of the calculation unit 23.
FIG. 14B is a diagram illustrating the calculation result of the positional relationship of the target image.

In the present embodiment, a target image having a close distance to the target image of interest is selectively extracted from all the target images displayed on the first image 61, and only the extracted target image is used to establish a relative positional relationship. Judge the change.

That is, the calculation unit 23 has a plurality of target images included in the first image 61 (point X), that is, the first image 81, the second image 82, the fifth image 85, the seventh image 87, and the eighth image 88. (Step S51). The calculation unit 23 calculates the relative positional relationship between the plurality of target images. As a result, the center point of the relative coordinate system is determined (step S52). For example, at the point X, the first image 81 of "Type B" closest to the center of the angle of view is determined as the target image of interest, and the position of the first image 81 is determined as the center point of the relative coordinate system.

Based on the first image 81 of "Type B", the second image 82 of "Type A", the fifth image 85 of "Type C", the seventh image 87 of "Type D", and " The relative positional relationship is calculated for each of the 8th image 88 of "Type E". The method of calculating this positional relationship is as described in FIGS. 6 (a) to 6 (d) described above. Further, a reference distance is set based on the size of the first image 81 and the like, and the first image 81 and the other images (second image 82, fifth image 85, seventh image 87, eighth image 88, respectively) are set. ) To find the distance.

As shown in FIG. 14B, in the relationship between the first image 81 of "Type B" and the second image 82 of "Type A", the second image 82 is the (left side, left side, Located on the upper side). Further, the distance between the first image 81 and the second image 82 is, for example, five times the reference distance. Therefore, this is expressed as (-, +5). Similarly, in the relationship between the first image 81 of "Type B" and the fifth image 85 of "Type C", it can be expressed as (+, +5). In the relationship between the first image 81 of "Type B" and the seventh image 87 of "Type D", it can be expressed as (-, +, 15). In the relationship between the first image 81 of "Type B" and the eighth image 88 of "Type E", it can be expressed as (+, +, 15).

Here, the calculation unit 23 selectively extracts only the target images (here, the second image 82 and the fifth image 85) located within a certain range from the position of the first image 81, and the extracted target images. Only the relative positional relationship by is the target of comparison. For example, when the distance threshold is set to "10", the relative positional relationship of the 7th image 87 of "Type D" and the relative positional relationship of the 8th image 88 of "Type E" are Not considered for comparison. That is, "Type D" and "Type E" are ignored. Therefore, in the second image 62 (point Z), the relative positional relationship of the ninth image 89 of "Type D" and the relative positional relationship of the tenth image 90 of "Type E" are calculated. It is not considered as a comparison target.

15 (a) and 15 (b) are diagrams illustrating the operation of the display control unit 24 according to the third embodiment.
FIG. 15A is a flowchart illustrating an operation example of the display control unit 24.
FIG. 15B is a diagram illustrating the determination result of the target image of interest.

As shown in FIG. 15A, the display control unit 24 determines whether or not there is a substantial change between the positional relationship at the point X (and the point Y) and the positional relationship at the point Z. (Step S61), the related information to be displayed is determined based on the determination result (step S62), and the display position of the related information is determined (step S63).

In this example, the first direction hd1 and the third direction hd3 are the same, and the second direction hd2 and the fourth direction hd4 are the same. Therefore, the state of change between the first positional relationship 51 and the second positional relationship 52 is determined to be the first state. Further, the fifth direction hd5 and the seventh direction hd7 are the same, and the sixth direction hd6 and the eighth direction hd8 are the same. Therefore, the state of change between the third positional relationship 53 and the fourth positional relationship 54 is determined to be the third state. In this case, the display control unit 24 determines that there is no substantial change in the positional relationship at the point Z, and continuously displays the related information displayed at the point X. On the other hand, when it is determined that the positional relationship has changed at the point Z, the target image of interest is set again, and the related information to be displayed is switched to another related information.

At points X and Y, it is assumed that the target image of "Type B" has been determined as the target image of interest. When arriving at the point Z, it is determined whether or not the relative positional relationship has substantially changed. Comparing the positional relationship at the point X (and the point Y) with the positional relationship at the point Z, it can be seen that these positional relationships have not changed substantially. Therefore, it can be determined that the target image of interest is a “Type B” target image like the point X (and the point Y). In this case, the related information to be displayed can be determined as the first related information 41, that is, "the device to be inspected first".

The display control unit 24 may adaptively determine the display position of the related information. For example, a position around the target image of interest and not overlapping with other detected target images can be determined as a display position.

16 (a) to 16 (c) are schematic views illustrating the display result of the related information according to the third embodiment.
FIG. 16A is a diagram illustrating a display result of related information at the point X.
FIG. 16B is a diagram illustrating the display result of the related information at the point Y.
FIG. 16C is a diagram illustrating a display result of related information at the point Z.

As shown in FIG. 13A, there are 71st to 5th objects 71 to 75 in the real space RS. The points X, Y, and Z in this example are the same as the points X, Y, and Z shown in FIG. 13 (b). For example, the information processing apparatus 112 displays the first related information 41 regarding the first object 71, and the worker 80 displays the first related information 41 on the first to fifth objects 71 to 75 while looking at the first related information 41. approach. The worker 80 moves in the order of point X, point Y, and point Z. The worker 80 is moving while holding the information processing device 112 over these plurality of objects.

As shown in FIG. 16A, the first image 61 (point X) includes the first image 81, the second image 82, the fifth image 85, and the seventh image 87 of the fourth object 74. Includes the eighth image 88 of the fifth object 75. However, the 7th image 87 and the 8th image 88 are not subject to comparison of relative positional relationships. Only the first image 81, the second image 82, and the fifth image 85 surrounded by the dotted line are subject to the comparison of the relative positional relationship. As shown in FIG. 16 (c), the second image 62 (point Z) includes only the third image 83, the fourth image 84, and the sixth image 86. That is, the second image 62 does not include the ninth image 89 of the fourth object 74 and the tenth image 90 of the fifth object 75, but the relative positional relationship of these object images is used as a comparison target. Not considered. Therefore, at the point Z, the first related information 41 displayed at the point X is continuously displayed as it is.

As described above, according to the embodiment, when many target images are displayed, the relative positional relationship can be calculated and compared by focusing on the target images having a short distance from the target image of interest. Therefore, the arithmetic processing can be executed quickly.

(Fourth Embodiment)
17 (a) and 17 (b) are schematic views showing an application example of the information processing apparatus according to the fourth embodiment.
FIG. 17A is a schematic diagram illustrating a plurality of target persons.
FIG. 17B is a schematic view illustrating a state in which the guide 80a holds the information processing device 113 and images a plurality of target persons.

The information processing device 113 is, for example, a tablet terminal provided with a camera (imaging unit 60). In this embodiment, there are a plurality of persons in the real space. On the other hand, it is assumed that the guide 80a holds the information processing device 113 and confirms the guidance instruction according to the person of interest.

As shown in FIG. 17A, in the real space RS, as a plurality of target persons, a first target person 91 of a “female under teen”, a second target person 92 of a “female in her thirties”, and “ There is a third target person 93 of "male in his 40s".

As shown in FIG. 17B, the information processing device 113 displays the first image 61 in which the first to third target persons 91 to 93 are captured. The first image 61 includes a first image 101 of the first target person 91, a second image 102 of the second target person 92, and a third image 103 of the third target person 93.

18 (a) and 18 (b) are diagrams illustrating the operation of the detection unit 21 according to the fourth embodiment.
FIG. 18A is a schematic diagram illustrating the first image 61.
FIG. 18B is a flowchart illustrating an operation example of the detection unit 21.

As shown in FIG. 18A, each target person's image includes a face. As shown in FIG. 18B, the detection unit 21 detects the face (step S71). There are various methods for face detection technology. For example, the presence or absence of a face is detected while scanning windows of various sizes in an image. According to this, it is possible to deal with faces of different sizes. For each window, the presence or absence of a face is determined using a co-occurrence pattern of luminance changes that expresses the structural features of the face. This makes it possible to suppress the effects of fluctuations in lighting conditions and differences in skin color.

The detection unit 21 detects the feature points of the face (step S72). From the detected face area, positions (feature points) representing structural features of the face such as eyes, nose, and mouth are extracted. By this facial feature point detection, the position of the target person image can be aligned and acquired.

The detection unit 21 recognizes the attribute (step S73). In attribute recognition, various attributes of the detected face are determined. Specific examples include gender and age. For each detected face, male or female is distinguished, and the age group is classified by teens. As a method for identifying the gender and age group, the facial feature points are associated with the referenced facial model based on the detected facial feature points. After correcting the direction of the face to the front, a dictionary is created from the face patterns collected in advance for each class, and the classes that are close to the dictionary are output as the classification result.

19 (a) and 19 (b) are diagrams illustrating the operation of the corresponding unit 22 according to the fourth embodiment.
FIG. 19A is a diagram illustrating related information 40a.
FIG. 19B is a flowchart illustrating an operation example of the corresponding unit 22.

As shown in FIG. 19A, it holds a database that associates the attribute type of the target person image with the related information 40a. This database is stored in advance in the first storage unit 40.

As shown in FIG. 19B, the corresponding unit 22 searches the database (first storage unit 40) based on the attribute type of each target person image (step S81), and provides related information to each target person image. Correspondence (step S82). Corresponding unit 22 can hold a database that associates the type of face attribute detected from the image with the related information. For example, it is assumed that the first image 101 of "a woman under teenager" is set as a person to be watched. If you refer to the database based on this, in order to meet the condition of "under teens", the fourth related information 44, that is, "The XXX event for children will be held in 30 minutes. Please guide me." Be attached.

FIG. 20 is a schematic diagram illustrating the display result of the related information according to the fourth embodiment.
As shown in FIG. 20, in the information processing device 113, the fourth related information 44, that is, the "XXX event for children" is held in the first image 101 of "women under teens" 30 minutes later. Please guide me. "Is displayed.

In this way, even when the face is targeted, it is possible to present the related information suitable for the attribute type of the person to the user (guide). Even in this case, the same related information can be continuously displayed as long as the relative positional relationship of the target person does not change. Therefore, it is possible to display the related information in an easy-to-read manner.

(Fifth Embodiment)
FIG. 21 is a block diagram illustrating an information processing apparatus according to a fifth embodiment.
As shown in FIG. 21, the information processing apparatus 114 according to the embodiment is further provided with an operation input unit 26.

The processing unit 20 further performs an operation of changing the target image of interest, which is a reference of the relative positional relationship, from the first image 81 to the second image 82. The operation of the change includes generating the fifth data d5 in which the second related information 42 (see FIG. 5A) associated with the second image 82 is superimposed and displayed on the first image 61. The processing unit 20 executes the change operation based on the operation input received by the operation input unit 26.

22 (a) to 22 (c) are diagrams showing an application example of the information processing apparatus according to the fifth embodiment.
FIG. 22A is a block diagram illustrating the operation input unit 26.
FIG. 22B is a schematic diagram showing a screen display example of the operation area.
FIG. 22C is a diagram illustrating the relationship between the operation direction and the target image.

The information processing device 114 is, for example, a tablet terminal provided with a camera (imaging unit 60). In the present embodiment, it is assumed that there are a plurality of objects in the real space and the worker 80 is holding the information processing device 114.

As shown in FIG. 22A, the operation input unit 26 includes an operation acquisition unit 26a, a relative coordinate calculation unit 26b, and a target image change unit 26c. The operation input unit 26 is based on the positions of a plurality of target images input from the detection unit 21, the target image of interest set by the setting unit 25, and the operation direction input to the operation input unit 26. And change the target image of interest. For example, when the first image 81 of "Type B" is set as the image of interest, the change of the image of interest is changed to the second image 82 of "Type A" or the fifth image 85 of "Type C". Carry out the process.

As shown in FIG. 22B, the operation acquisition unit 26a detects the drag direction by the operator 80 in the operation area 26d provided on the display screen of the information processing device 114 and acquires the operation direction. Here, it is assumed that the operation is performed on the upper right.

The relative coordinate calculation unit 26b determines which target in which direction based on the relative coordinates (first positional relationship 51, third positional relationship 53) starting from the target image of interest (first image 81) obtained by the calculation unit 23. Ask if the statue exists. In this example, the operation direction is divided along the X-axis and the Y-axis, and the positions of the four quadrants, that is, (-,-), (-, +), (+, +), (+,-), respectively. Identify the target image that exists in. The calculation result is shown in FIG. 22 (c).

The target image changing unit 26c changes the target image of interest by associating the operation direction acquired by the operation acquisition unit 26a with the target image specified by the relative coordinate calculation unit 26b. Here, the target image of interest is changed to the fifth image 85 of "Type C" located at the upper right (relative coordinates (+, +)) of the first image 81. Alternatively, the image of interest is changed to the second image 82 of "Type A" located at the upper left (relative coordinates (-, +)) of the first image 81.

23 (a) to 23 (c) are schematic views illustrating the display results of the related information according to the fifth embodiment.
FIG. 23A is a diagram illustrating a display result of related information before the operation.
FIG. 23B is a diagram illustrating a screen during operation.
FIG. 23 (c) is a diagram illustrating a display result of related information after the operation.

As shown in FIGS. 23 (a) to 23 (c), when the operation input unit 26 changes the image of interest to the fifth image 85, the display control unit 24 is associated with the fifth image 85. The third related information 43 is superimposed on the first image 61 and displayed.

On the other hand, when the operation input unit 26 changes the target image of interest to the second image 82, the display control unit 24 superimposes the second related information 42 associated with the second image 82 on the first image 61. To display.

The method of selecting the target image of interest by the operation input unit 26 includes a method of directly touching the target image on the screen to select the target image of interest, a method of selecting the target image of interest by a toggle operation, and a plurality of methods in the same direction. Even if there is an target image of, various realization means such as a method of selecting the target image of interest according to the length of the drag operation can be considered.

As described above, according to the embodiment, by providing the operation input unit for the operator to independently change the image of interest, the display content of the related information can be switched at the timing when the continuous display of the related information is desired to be completed. it can. This makes it possible to display related information in an easy-to-read manner.

(Sixth Embodiment)
FIG. 24 is a block diagram illustrating an information processing apparatus according to a sixth embodiment.
The information processing device 115 according to the embodiment includes a desktop or laptop type general-purpose computer, a portable general-purpose computer, other portable information devices, an information device having an imaging device, a smartphone, another information processing device, and the like. It can be realized by various devices.

As shown in FIG. 24, the information processing apparatus 115 of the embodiment has a CPU 201, an input unit 202, an output unit 203, a RAM 204, a ROM 205, an external memory interface 206, and a communication interface as hardware configuration examples. 207 and.

The instructions given in the processing procedure shown in the above embodiments can be executed based on a program that is software. By storing this program in advance in a general-purpose computer system and reading this program, it is possible to obtain an effect similar to the effect of the information processing apparatus of the above-described embodiment. The instructions described in the above-described embodiments are the programs that can be executed by the computer, such as a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD). It is recorded on a recording medium (± R, DVD ± RW, etc.), a semiconductor memory, or a similar recording medium. The storage format may be any form as long as it is a recording medium that can be read by a computer or an embedded system. The computer can realize the same operation as the information processing apparatus of the above-described embodiment by reading the program from the recording medium and executing the instruction described in the program based on the program on the CPU. Of course, if the computer acquires or loads the program, it may acquire or load it through the network.

In addition, an OS (operating system) running on a computer based on instructions of a program installed on a computer or an embedded system from a recording medium, database management software, MW (middleware) running on a network, or the like is an embodiment. A part of each process for realizing it may be executed.

Further, the recording medium in the embodiment is not limited to a recording medium independent of a computer or an embedded system, but also includes a recording medium in which a program transmitted by a LAN, the Internet, or the like is downloaded and stored or temporarily stored. Further, the recording medium is not limited to one, and the case where the processing in the embodiment is executed from a plurality of recording media is also included in the recording medium in the embodiment. The structure of the recording medium may be any structure.

The computer or embedded system in the embodiment is for executing each process in the embodiment based on the program stored in the recording medium, and is a device including one such as a personal computer and a microcomputer, or , A system in which a plurality of devices are connected to a network, or the like may be used.

Further, the computer in the embodiment is not limited to a personal computer, but also includes an arithmetic processing device, a microcomputer, etc. included in an information processing device, and is a general term for devices and devices capable of realizing the functions in the embodiment by a program. ing.

According to the embodiment, it is possible to provide an information processing device, an information processing method, and an information processing program capable of easily displaying related information related to an object.

The embodiments of the present invention have been described above with reference to specific examples. However, the present invention is not limited to these specific examples. For example, regarding the specific configuration of each element such as the acquisition unit and the processing unit, the present invention can be similarly carried out by appropriately selecting from a range known to those skilled in the art, and as long as the same effect can be obtained. It is included in the scope of the invention.

Further, a combination of any two or more elements of each specific example to the extent technically possible is also included in the scope of the present invention as long as the gist of the present invention is included.

In addition, based on the above-mentioned information processing device, information processing method, and information processing program as an embodiment of the present invention, all information processing devices, information processing methods, and information processing that can be appropriately designed and implemented by those skilled in the art. The program also belongs to the scope of the present invention as long as it includes the gist of the present invention.

In addition, within the scope of the idea of the present invention, those skilled in the art can come up with various modified examples and modified examples, and it is understood that these modified examples and modified examples also belong to the scope of the present invention. ..

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

10 ... Acquisition unit, 20 ... Processing unit, 21 ... Detection unit, 22 ... Corresponding unit, 23 ... Calculation unit, 24 ... Display control unit, 25 ... Setting unit, 26 ... Operation input unit, 26a ... Operation acquisition unit, 26b ... Relative coordinate calculation unit, 26c ... Target image change unit, 26d ... Operation area, 30 ... Display unit, 40 ... First storage unit, 40a ... Related information, 41-44 ... First to fourth related information, 50 ... Second Storage unit, 50a ... Positional relationship, 51-54 ... 1st to 4th positional relationships, 60 ... Imaging unit, 61, 62 ... 1st and 2nd images, 61a ... Center coordinates, 71-75 ... 1st to 5th Object, 80 ... worker, 80a ... guide, 81-90 ... 1st to 10th images, 91-93 ... 1st to 3rd target persons, 101-103 ... 1st to 3rd images, 110-115 ... Information processing device, 201 ... CPU, 202 ... Input unit, 203 ... Output unit, 204 ... RAM, 205 ... ROM, 206 ... External memory interface, 207 ... Communication interface, a1 to a4 ... 1st to 4th axes, b1 , B2 ... 1st and 2nd positions, c1, c2 ... 1st and 2nd relative coordinate systems, d1 to d5 ... 1st to 5th data, hd1 to hd8 ... 1st to 8th directions, t1, t2 ... 1, 2nd time, RS ... Real space

Claims (12)

The first image at the first time, the first image including the first image of the first object and the second image of the second object, and the second image at the second time after the first time. An acquisition unit for acquiring the third image of the first object and the second image including the fourth image of the second object unlike the third image.
The first operation of generating the first data in which the first related information associated with the first image is superimposed on the first image and displayed.
The relative first positional relationship of the second image when the first image is used as a reference in the first image, and the third image when the third image is used as a reference in the second image. When there is no change between the relative second positional relationships of the four images, or when the first state is smaller than the change between the first positional relationship and the second positional relationship in the second state. In the second state, where the first related information is superimposed on the second image to generate second data, and there is a change between the first positional relationship and the second positional relationship. The second operation, in which the first related information is not superimposed on the second image and displayed, or another related information different from the first related information is superimposed and displayed on the second image.
And the processing department that implements
With
The processing unit
The operation of calculating the first positional relationship based on the first relative coordinate system centered on the first position in the first image, and
The operation of calculating the second positional relationship based on the second relative coordinate system centered on the second position in the third image, and
Further implemented,
The first relative coordinate system includes a first axis that passes through the first position and extends in a first direction, and a second axis that passes through the first position and extends in a second direction that intersects the first direction.
The first positional relationship includes a first direction along the first axis in the third direction from the first image to the second image and a second direction along the second axis in the third direction. Including
The second relative coordinate system includes a third axis that passes through the second position and extends in the first direction, and a fourth axis that passes through the second position and extends in the second direction.
The second positional relationship includes a third direction along the third axis in the fourth direction from the third image to the fourth image and a fourth direction along the fourth axis in the fourth direction. Including
The processing unit further performs an operation of comparing the first direction and the third direction, comparing the second direction and the fourth direction, and determining whether or not it is in the first state.
In the first state, the first orientation and the third orientation are the same, and the second orientation and the fourth orientation are the same.
In the second state, the information processing device in which the first orientation and the third orientation are different, or the second orientation and the fourth orientation are different. The first operation further includes an operation of recognizing the first image and the second image.
The second operation further includes an operation of recognizing the third image and the fourth image.
The information processing device according to claim 1. The information processing device according to claim 2, wherein the recognition includes recognition of at least one of a character, an object shape, and a human face. The information processing device according to any one of claims 1 to 3, wherein the processing unit further performs an operation of deriving the first positional relationship. In the operation of the derivation, the distance between the center of the first image and the center of the first image is shorter than the distance between the center of the second image and the center of the first image for a predetermined time. The information processing apparatus according to claim 4, wherein the first positional relationship is sometimes derived. The second image is located within a certain range from the position of the first image.
The information processing device according to any one of claims 1 to 5, wherein the fourth image is located within the range from the position of the third image. The first image further includes a fifth image of the third object.
The second image, unlike the third image and the fourth image, further includes a sixth image of the third object.
The first operation further includes generating third data in which the first related information is superimposed and displayed on the first image.
In the second operation, the state of the change is the first state, and the relative third positional relationship of the fifth image when the first image is used as a reference and the third image as a reference. When the relative fourth positional relationship of the sixth image and the state of change between them are the third state, the first related information is superimposed and displayed on the second image. The information processing apparatus according to any one of claims 1 to 6, further comprising generating data. The information processing apparatus according to any one of claims 1 to 7, wherein the processing unit further performs an operation of changing the reference from the first image to the second image. The information processing apparatus according to claim 8, wherein the operation of the change further includes generating fifth data in which the second related information associated with the second image is superimposed and displayed on the first image. .. Equipped with an operation input unit
The information processing device according to claim 8 or 9, wherein the processing unit executes the operation of the change based on the operation input received by the operation input unit. The first image at the first time, the first image including the first image of the first object and the second image of the second object, and the second image at the second time after the first time. Therefore, the third image of the first object and the second image including the fourth image of the second object, which is different from the third image, are acquired.
The first data in which the first related information associated with the first image is superimposed on the first image and displayed is generated.
The relative first positional relationship of the second image when the first image is used as a reference in the first image, and the third image when the third image is used as a reference in the second image. When there is no change between the relative second positional relationships of the four images, or when the first state is smaller than the change between the first positional relationship and the second positional relationship in the second state. In the second state, where the first related information is superimposed on the second image to generate second data, and there is a change between the first positional relationship and the second positional relationship. , The first related information is not superimposed on the second image and displayed, or another related information different from the first related information is superimposed and displayed on the second image.
The operation of calculating the first positional relationship based on the first relative coordinate system centered on the first position in the first image, and
The operation of calculating the second positional relationship based on the second relative coordinate system centered on the second position in the third image, and
And carry out
The first relative coordinate system includes a first axis that passes through the first position and extends in a first direction, and a second axis that passes through the first position and extends in a second direction that intersects the first direction.
The first positional relationship includes a first direction along the first axis in the third direction from the first image to the second image and a second direction along the second axis in the third direction. Including
The second relative coordinate system includes a third axis that passes through the second position and extends in the first direction, and a fourth axis that passes through the second position and extends in the second direction.
The second positional relationship includes a third direction along the third axis in the fourth direction from the third image to the fourth image and a fourth direction along the fourth axis in the fourth direction. Including
An operation of comparing the first direction and the third direction, comparing the second direction and the fourth direction, and determining whether or not the state is the first state is performed.
In the first state, the first orientation and the third orientation are the same, and the second orientation and the fourth orientation are the same.
In the second state, the information processing method in which the first orientation and the third orientation are different, or the second orientation and the fourth orientation are different. The first image at the first time, the first image including the first image of the first object and the second image of the second object, and the second image at the second time after the first time. The process of acquiring the third image of the first object and the second image including the fourth image of the second object, which is different from the third image.
A process of generating first data in which the first related information associated with the first image is superimposed on the first image and displayed.
The relative first positional relationship of the second image when the first image is used as a reference in the first image, and the third image when the third image is used as a reference in the second image. When there is no change between the relative second positional relationships of the four images, or when the first state is smaller than the change between the first positional relationship and the second positional relationship in the second state. In the second state, where the first related information is superimposed on the second image to generate second data, and there is a change between the first positional relationship and the second positional relationship. , The process of not displaying the first related information superimposed on the second image, or displaying another related information different from the first related information superimposed on the second image.
The operation of calculating the first positional relationship based on the first relative coordinate system centered on the first position in the first image and the second relative coordinate system centered on the second position in the third image. A process including the operation of calculating the second positional relationship based on the above, wherein the first relative coordinate system has a first axis extending in the first direction through the first position and the first position. The first positional relationship includes the second axis extending in the second direction intersecting the first direction, and the first positional relationship is along the first axis in the third direction from the first image to the second image. The second relative coordinate system includes a first direction and a second direction along the second axis in the third direction, and the second relative coordinate system includes a third axis extending in the first direction through the second position and the first direction. The second positional relationship includes a fourth axis that passes through two positions and extends in the second direction, and the second positional relationship is a third direction along the third axis in the fourth direction from the third image to the fourth image. , A fourth orientation along the fourth axis of the fourth orientation, and processing.
The first state is a process including an operation of comparing the first direction and the third direction, comparing the second direction and the fourth direction, and determining whether or not the state is the first state. Then, the first orientation and the third orientation are the same, the second orientation and the fourth orientation are the same, and in the second state, the first orientation and the third orientation are different. that, or the second direction and said fourth direction are different, and processing,
An information processing program that lets a computer execute.

Images