WO2021235193A1

WO2021235193A1 - Information processing system, information processing method and program

Info

Publication number: WO2021235193A1
Application number: PCT/JP2021/016735
Authority: WO
Inventors: 保乃花尾崎; 健太郎井田; 拓也池田
Original assignee: ソニーグループ株式会社
Priority date: 2020-05-21
Filing date: 2021-04-27
Publication date: 2021-11-25
Also published as: JPWO2021235193A1

Abstract

This information processing system (1, 2) comprises an error information detection unit (16) and a correction unit (17). The error information detection unit (16) detects information about an error that occurred in the relative positions of a plurality of local coordinate systems (LC). The correction unit (17), in response to the start of a content (CT) operation, corrects the relative positions of all other local coordinate systems (LC) in terms of a specified local coordinate system (LC) on the basis of the information about the error. The specified local coordinate system (LC) is the local coordinate system (LC), out of the plurality of local coordinate systems (LC), that is used to generate content (CT) which an operation subject of the content (CT) recognizes as an operation target.

Description

Information processing systems, information processing methods and programs

The present invention relates to an information processing system, an information processing method and a program.

AR (Augmented Reality) technology that expands the real environment with a computer is attracting attention. AR technology is realized by devices such as AR glasses, head-mounted displays, and projectors that superimpose and display content in real space.

Japanese Unexamined Patent Publication No. 2011-203823

The position of the content superimposed on the real space changes depending on the position and posture of the device. If an error occurs in the detection result of the sensor that detects the position and orientation of the device, the coordinate system (origin position and coordinate axis position) set in the device will be deviated. When a plurality of devices share one AR space, an error occurs in the relative position between the coordinate systems set for each device, and the position of the content displayed on each device is deviated. Therefore, it is desirable to correct the coordinate system of each device to align the position of the content. However, if the position of the content is corrected during the operation of the content, the operation subject of the content may feel uncomfortable.

Therefore, in this disclosure, we propose an information processing system, information processing method, and program that does not cause discomfort to the operating subject when correcting the coordinate system.

According to the present disclosure, an error information detection unit that detects information on an error that has occurred in a relative position of a plurality of local coordinate systems, and a plurality of local coordinate systems in response to the start of operation of the content. A correction unit that corrects the relative positions of all other local coordinate systems with respect to the specific local coordinate system used to generate the content, which the operating subject of the content recognizes as an operation target, based on the error information. The information processing system to have is provided. Further, according to the present disclosure, an information processing method in which the information processing of the information processing system is executed by a computer, and a program for realizing the information processing of the information processing system in the computer are provided.

It is a schematic diagram of the information processing system of 1st Embodiment. It is a figure which shows an example of the coordinate system set in the real space. It is a figure explaining an example of the correction method of a local coordinate system. It is a figure explaining an example of the correction method of a local coordinate system. It is a figure explaining an example of the correction method of a local coordinate system. It is a figure explaining an example of the correction method of a local coordinate system. It is a figure which shows the example which applied the information processing system to the exhibition of a city model. It is a figure which shows an example of the content from the viewpoint of the 2nd user. It is a figure which shows the correction method of the relative position of a glass content and PJ content. It is a figure which shows the correction method of the relative position of a glass content and PJ content. It is a figure which shows the other application example of an information processing system. It is a figure which shows the other application example of an information processing system. It is a figure which shows the other application example of an information processing system. It is a flowchart which shows the information processing method. It is a schematic diagram of the information processing system of the 2nd Embodiment. It is a figure which shows the example which applied the information processing system to the correction of a robot coordinate system. It is a figure which shows the example which applied the information processing system to the correction of a robot coordinate system.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

The explanations are given in the following order.
[1. First Embodiment]
[1-1. Information processing system configuration]
[1-2. Information processing method]
[1-3. effect]
[2. Second Embodiment]
[2-1. Information processing system configuration]
[2-2. effect]
[2-3. Modification example]

[1. First Embodiment]
[1-1. Information processing system configuration]
FIG. 1 is a schematic diagram of the information processing system 1 of the first embodiment.

The information processing system 1 includes, for example, a processing device 10, a plurality of devices 20, and a storage device 30. The processing device 10 generates the content CT displayed by the plurality of devices 20. The content CT is superimposed and displayed on the real space RS by the device 20.

The plurality of devices 20 include, for example, an AR glass 20A, an AR glass 20B, and a projector 20C. The AR glasses 20A and the AR glasses 20B are attached to the head of the user U, and the content CT is superimposed and displayed on the real space RS in the field of view of the user U. The projector 20C is fixed at a predetermined position in the real space RS and projects the content CT onto the screen SCR provided in the real space RS. The screen SCR is, for example, a table.

FIG. 2 is a diagram showing an example of a coordinate system set in the real space RS.

A local coordinate system LC is set for each of the plurality of devices 20. The local coordinate system LC has an origin and coordinate axes defined for each device 20. The origin position and the position of the coordinate axis of the local coordinate system LC are calculated based on information such as the internal parameters for each device 20 included in the registration data 32, the installation position, and the posture at the installation position. The processing device 10 calculates the coordinates of the content CT displayed on the device 20 by using the local coordinate system LC set on the device 20.

Hereinafter, the content CT displayed by the AR glass 20A and the AR glass 20B will be referred to as a glass content GC. The content CT displayed by the projector 20C is referred to as PJ content. The local coordinate system LC set in the AR glass 20A is referred to as a glass coordinate system LCA. The local coordinate system LC set in the AR glass 20B is referred to as a glass coordinate system LCB. The local coordinate system LC set in the projector 20C is referred to as a PJ coordinate system LCC.

In FIG. 2, the reference numeral WC indicates a world coordinate system. The world coordinate system WC is a coordinate system that defines the entire real space RS. The world coordinate system WC is defined, for example, based on the results of a pre-performed real-space RS scan. The PJ coordinate system LCC is defined based on the detection result of a sensor fixed at a predetermined position in the real space RS. The glass coordinate system LCA is estimated from the detection result of the sensor mounted on the AR glass 20A. The glass coordinate system LCB is estimated from the detection result of the sensor mounted on the AR glass 20B.

The world coordinate system WC is the coordinate system most conforming to the real world. Therefore, the correction of the positional deviation generated in the content CT of each device 20 may be performed with reference to the world coordinate system WC. However, in this embodiment, in order to reduce the amount of calculation, the PJ coordinate system LCC is used as the reference local coordinate system, and the glass coordinate system LCA and the glass coordinate system LCB are used as the reference local coordinate system without using the world coordinate system WC. A relative position correction method of matching is used. This is because the sensor of the projector 20C installed at the fixed position is considered to have a smaller detection error when used for a long time than the sensors of the AR glasses 20A and the AR glasses 20B that move with the user U. be.

Hereinafter, the correction method of the content CT will be described in detail in accordance with the specific rules.

In the example of FIG. 1, chess is performed by the first user U1 wearing the AR glass 20A and the second user U2 wearing the AR glass 20B. The AR glass 20A and the AR glass 20B display a virtual object representing a chess piece as a glass content GC. The projector 20C displays a virtual object representing a chess board as PJ content PJC. The glass content GC is displayed according to the squares of the PJ content PJC.

Each device 20 has one or more sensors. Each device 20 outputs the sensor data detected by one or more sensors to the processing device 10. For example, the AR glass 20A and the AR glass 20B have a plurality of sensors including a depth sensor, an accelerometer, a microphone and a camera. The sensor data output from the AR glasses 20A and the AR glasses 20B is used, for example, for recognizing the real space RS, estimating the position in the real space RS, and detecting the gesture of the user U. The projector 20C has a plurality of sensors including a depth sensor, a motion sensor, a microphone and a camera. The sensor data output from the projector 20C is used, for example, to detect the position of the user U in the real space RS.

The processing device 10 is, for example, a glass sensor data acquisition unit 11, a PJ sensor data acquisition unit 12, a space recognition unit 13, a gesture detection unit 14, an interaction detection unit 15, an error information detection unit 16, a correction unit 17, and a content generation unit 18. And has a display control unit 19.

The glass sensor data acquisition unit 11 acquires, for example, the sensor data output from the AR glass 20A and the AR glass 20B. The glass sensor data acquisition unit 11 outputs, for example, the acquired sensor data to the space recognition unit 13 and the gesture detection unit 14. The PJ sensor data acquisition unit 12 acquires, for example, the sensor data output from the projector 20C. The PJ sensor data acquisition unit 12 outputs, for example, the acquired sensor data to the interaction detection unit 15.

The space recognition unit 13 recognizes the real space RS around the AR glass 20A and the AR glass 20B, and the AR glass 20A and the AR glass in the real space RS, for example, based on the sensor data acquired from the glass sensor data acquisition unit 11. The position of 20B is estimated, and the real objects around the AR glasses 20A and AR glasses 20B are detected. Recognition of the real space RS and estimation of the position in the real space RS are performed, for example, by using a technique called SLAM (Simultaneus Localization and Mapping). The space recognition unit 13 outputs, for example, information on the glass coordinate system (information on the position of the origin and the position of the coordinate axis) generated using SLAM and information on the environment map to the error information detection unit 16.

The gesture detection unit 14 detects the gestures of the first user U1 and the second user U2 based on the sensor data acquired from the glass sensor data acquisition unit 11, for example. For example, the gesture detection unit 14 detects the gesture based on the movement of the operation part OP (the part where the operation subject interacts with the content CT) of the user U who is the operation subject of the content CT.

The operation site OP is a site where the position or form of the content CT can be changed by the movement. For example, when the operation of the content CT is performed by the movement of the hand or finger of the user U in contact with the content CT, the operation site OP is the hand or finger of the user U in contact with the content CT. When the operation of the content CT is performed by the movement of the hand or finger of the user U separated from the content CT by a predetermined distance (when moving, rotating, or deforming a virtual object such as a ball without contact), the operation site OP is , The hand or finger of the user U separated from the content CT by a predetermined distance.

The gesture detection unit 14 identifies the user U who operates the content CT, for example, based on the sensor data acquired from the glass sensor data acquisition unit 11. The gesture detection unit 14 outputs, for example, information about the operation subject of the content CT and the gesture for the content CT performed by the operation subject to the content generation unit 18.

The gesture detection unit 14 detects, for example, the coordinates of the operation site OP in the glass coordinate system LCA and the glass coordinate system LCB, respectively, based on the sensor data acquired from the glass sensor data acquisition unit 11. The gesture detection unit 14 outputs the coordinate information of the operation site OP detected for each glass coordinate system to the error information detection unit 16. The gesture detection unit 14 outputs a detection error signal to the error information detection unit 16, for example, when the operation subject of the content CT cannot be detected or when the coordinates of the operation site OP cannot be detected.

The interaction detection unit 15 detects the coordinates of the operation site OP in the PJ coordinate system LCC, for example, based on the sensor data acquired from the PJ sensor data acquisition unit 12. The interaction detection unit 15 outputs, for example, the coordinate information of the operation site OP in the PJ coordinate system LCC to the error information detection unit 16.

The interaction detection unit 15 determines, for example, the accuracy of the sensor data acquired from the PJ sensor data acquisition unit 12. The accuracy of the sensor data is quantified according to, for example, the distance between the sensor and the user U, the size of the operation site OP superimposed on the content CT, and the like. For example, when the user U is located at a position far away from the sensor of the projector 20C, or when occlusion occurs with respect to the sensor of the projector 20C, the accuracy of the calculated sensor data is low. For example, when the accuracy of the sensor data is equal to or less than the threshold value, the interaction detection unit 15 outputs a detection error signal to the error information detection unit 16.

The error information detection unit 16 detects information on errors that occur at relative positions of a plurality of local coordinate systems LC. The error information detection unit 16 detects, for example, the deviation generated at the relative positions of the origin and the coordinate axes of the plurality of local coordinate systems LC as error information.

The error information detection unit 16 detects error information based on the coordinates of the operation site OP detected for each local coordinate system LC, for example. For example, as an initial setting, it is assumed that the origin positions and coordinate axis positions of all the local coordinate system LCs are calibrated so as to match. In the initial state immediately after calibration, the coordinates of the operation site OP in the PJ coordinate system LCC and the coordinates of the operation site OP in the glass coordinate system LCA match. After that, it is assumed that the coordinates of the operation site OP in the PJ coordinate system LCC become (30, 40, 50) and the coordinates of the operation site OP in the glass coordinate system LCA become (25, 45, 55) due to long-term use. .. In this case, the error generated in the relative position between the PJ coordinate system LCC and the glass coordinate system LCA is calculated as (-5, 5, 5).

The error information detection unit 16 outputs, for example, error information to the correction unit 17. For example, when the error information detection unit 16 acquires a detection error signal from the gesture detection unit 14 and the interaction detection unit 15, the error information detection unit 16 outputs the correction error signal to the correction unit 17.

The correction unit 17 corrects, for example, the relative positions of all other local coordinate system LCs with respect to the specific local coordinate system LC based on the error information in response to the start of the operation of the content CT. For example, when the correction unit 17 acquires a correction error signal from the error information detection unit 16, the correction unit 17 does not correct the local coordinate system LC.

The specific local coordinate system LC is the local coordinate system LC used to generate the content CT that the operating subject of the content CT recognizes as an operation target. By the above-mentioned correction, for example, the origin position and the position of the coordinate axis of the local coordinate system LC other than the specific local coordinate system LC are moved to the position where the error of the relative position with the specific local coordinate system LC becomes small. It is preferred that the origins and axes of the other local coordinate system LCs move to positions where the relative position errors that occur with the particular local coordinate system LC are offset.

Hereinafter, an example of the correction method of the local coordinate system LC will be described with reference to FIGS. 3 to 6.

FIG. 3 is a diagram showing the arrangement of each local coordinate system LC after a predetermined time has elapsed from the initial state. The dotted line shows the arrangement of the local coordinate system LC in the initial state. The solid line shows the arrangement of the local coordinate system LC after the lapse of a predetermined time. The glass coordinate system LCA is in a state of being rotated by the rotation amount RT in the first direction D1 as compared with the initial state. The glass coordinate system LCB is in a state of being translated by a distance PM in the second direction D2 as compared with the initial state. The position of the PJ coordinate system LCC has not changed.

FIG. 4 shows an example in which the first user U1 performs an operation of moving a piece to the glass content GC as the first case. In this case, the operating subject is the first user U1. The content CT recognized by the operation subject as the operation target is the glass content GC generated by using the glass coordinate system LCA. When the first user U1 starts the movement operation of the glass content GC, the positions of the glass coordinate system LCB and the PJ coordinate system LCC are corrected with reference to the glass coordinate system LCA. For example, the origin position and the position of the coordinate axis of the glass coordinate system LCB and the PJ coordinate system LCC move to a position where the error of the relative position with the glass coordinate system LCA becomes small. In the example of FIG. 4, the glass coordinate system LCB is corrected by rotating in the first direction D1 by the rotation amount RT and translating in the third direction D3 opposite to the second direction D2 by the distance PM. For the PJ coordinate system LCC, a correction is performed to rotate the PJ coordinate system LCC by the rotation amount RT in the first direction D1.

FIG. 5 shows, as a second case, a correction method when the second user U2 performs an operation of moving a piece with respect to the glass content GC. In this case, the operating subject is the second user U2. The content CT recognized by the operation subject as the operation target is the glass content GC generated by using the glass coordinate system LCB. When the second user U2 starts the movement operation of the glass content GC, the positions of the glass coordinate system LCA and the PJ coordinate system LCC are corrected with reference to the glass coordinate system LCB. For example, the origin position and the position of the coordinate axis of the glass coordinate system LCA and the PJ coordinate system LCC move to a position where the error of the relative position with the glass coordinate system LCB becomes small. In the example of FIG. 5, the glass coordinate system LCA is corrected by rotating in the fourth direction D4 opposite to the first direction D1 by the rotation amount RT and translating in the second direction D2 by the distance PM. For the PJ coordinate system LCC, correction is performed to translate the PJ coordinate system LCC by the distance PM in the second direction D2.

User U can operate not only the glass content GC but also the PJ content PJC. FIG. 6 shows, as a third case, a correction method when the PJ content PJC includes a design change menu for changing the design of the board and the first user U1 operates the design change menu. In this case, the operating subject is the first user U1. The content CT recognized by the operation subject as the operation target is the PJ content PJC generated by using the PJ coordinate system LCC. When the first user U1 starts the operation of the design change menu, the positions of the glass coordinate system LCA and the glass coordinate system LCB are corrected with reference to the PJ coordinate system LCC. For example, the origin position and the position of the coordinate axis of the glass coordinate system LCA and the glass coordinate system LCB move to a position where the error of the relative position with the PJ coordinate system LCC becomes small. In the example of FIG. 6, the glass coordinate system LCA is corrected to rotate by the rotation amount RT in the fourth direction D4. For the glass coordinate system LCB, correction is performed to translate the glass coordinate system LCB by the distance PM in the third direction D3.

For example, the correction unit 17 ends the correction based on the specific local coordinate system LC in response to the completion of the operation of the content CT. Then, the correction unit 17 switches the local coordinate system LC that is the reference of the correction from the specific local coordinate system LC to the reference local coordinate system (PJ coordinate system LCC). The correction unit 17 corrects the relative positions of all other local coordinate system LCs with respect to the reference local coordinate system based on the error information. The reference local coordinate system is a local coordinate system LC in which an error generated in a position relative to the world coordinate system WC is smaller than that of a specific local coordinate system LC. As described above, in the glass coordinate system, the origin position and the position of the coordinate axis are more likely to shift due to changes in the environment or long-term use as compared with the PJ coordinate system LCC. Therefore, the PJ coordinate system LCC is used as the reference local coordinate system.

By this correction, for example, the origin position and the position of the coordinate axis of the local coordinate system LC other than the reference local coordinate system move to the position where the error of the relative position with the reference local coordinate system becomes small. It is preferable that the origin position and the position of the coordinate axis of the other local coordinate system LC are moved to a position where the error of the relative position generated with the reference local coordinate system is offset.

For example, in the first case described above, when the first user U1 finishes the movement operation of the glass content GC, the correction based on the glass coordinate system LCA is completed, and the local coordinate system LC as the reference of the correction is the PJ coordinate system. Switch to LCC (reference local coordinate system). Then, the positions of the glass coordinate system LCA and the glass coordinate system LCB are corrected with reference to the PJ coordinate system LCC. For example, the origin position and the position of the coordinate axis of the glass coordinate system LCA and the glass coordinate system LCB move to a position where the error of the relative position with the PJ coordinate system LCC becomes small. In the example of FIG. 3, the position of the PJ coordinate system LCC has not changed from the initial state. Therefore, the glass coordinate system LCA and the glass coordinate system LCB are returned to the positions in the initial state shown by the dotted line by the correction based on the PJ coordinate system LCC. The same applies to the second case described above.

In the third case described above, the local coordinate system LC used as the reference for correction when the first user U1 starts the operation of the PJ content PJC is the PJ coordinate system LCC (reference local coordinate system). Therefore, even if the first user U1 finishes the operation of the PJ content PJC, the correction based on the PJ coordinate system LCC is maintained.

Returning to FIG. 1, the correction unit 17 generates, for example, a first coordinate conversion formula that performs coordinate conversion between the glass coordinate system LCA and the PJ coordinate system LCC. The correction unit 17 generates, for example, a second coordinate conversion formula that performs coordinate conversion between the glass coordinate system LCB and the PJ coordinate system LCC. The correction unit 17 outputs the first coordinate conversion formula and the second coordinate conversion formula to the content generation unit 18.

The content generation unit 18 generates, for example, a content CT that each device 20 superimposes on the real space RS and displays it, and outputs the content CT to the display control unit 19. The content generation unit 18 generates, for example, a content CT corresponding to the gesture detected by the gesture detection unit 14. The content generation unit 18 sets the coordinates of the content CT using, for example, the PJ coordinate system LCC in which the origin position and the position of the coordinate axis are least likely to deviate. The content generation unit 18 calculates the coordinates of the content CT in the glass coordinate system LCA and the glass coordinate system LCB by using the first coordinate conversion formula and the second coordinate conversion formula.

The display control unit 19 outputs, for example, the content CT generated by the content generation unit 18 to each device 20. The display control unit 19 adds a correction process to the content CT generated by the content generation unit 18 as necessary. For example, the projector 20C projects the content CT onto the screen SCR. Therefore, the display control unit 19 outputs the content CT subjected to the geometric correction based on the screen model included in the registration data 32 to the projector 20C. The screen model is a three-dimensional model of the screen SCR. The screen model contains, for example, coordinate information on the surface of the screen SCR on which the content CT is projected.

For example, the display control unit 19 gradually changes the position of the content CT to the position calculated by the correction based on the reference local coordinate system in response to the completion of the operation of the content CT. This reduces the sense of discomfort caused by the sudden change in the position of the content CT after the operation is completed.

The storage device 30 stores, for example, the program 31 executed by the processing device 10 and the registration data 32. The program 31 is a program that causes a computer to execute information processing according to the present embodiment. The processing device 10 performs various processes according to the program 31 stored in the storage device 30. The storage device 30 may be used as a work area for temporarily storing the processing result of the processing device 10. The storage device 30 includes any non-transient storage medium such as, for example, a semiconductor storage medium and a magnetic storage medium. The storage device 30 includes, for example, an optical disk, a magneto-optical disk, or a flash memory. The program 31 is stored, for example, in a non-transient storage medium that can be read by a computer.

The processing device 10 is, for example, a computer composed of a processor and a memory. The memory of the processing device 10 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). By executing the program 31, the processing device 10 executes the glass sensor data acquisition unit 11, the PJ sensor data acquisition unit 12, the space recognition unit 13, the gesture detection unit 14, the interaction detection unit 15, the error information detection unit 16, and the correction unit. 17. Functions as a content generation unit 18 and a display control unit 19.

FIG. 7 is a diagram showing an example in which the information processing system 1 is applied to an exhibition of a city model.

In the example of FIG. 7, the stereoscopic image of the building is superimposed and displayed on the planar image of the city model projected on the screen SCR. The plane image of the city model projected on the screen SCR is the content CT (PJ content PJC) displayed by the projector 20C. The stereoscopic image of the building superimposed on the planar image of the city model is the content CT (glass content GC) displayed by the AR glass 20A and the AR glass 20B.

FIG. 8 is a diagram showing an example of content CT (PJ content PJC and glass content GC) viewed from the viewpoint of the second user U2.

The glass content GC is a three-dimensional display of a part of the building BLD displayed on the PJ content PJC based on the viewpoint of the second user U2. The glass content GC is superimposed on the area OVL in which the plan image of the building BLD is displayed in the PJ content PJC. FIG. 7 shows how the second user U2 reaches for the glass content GC and tries to operate the glass content GC with his / her finger.

As described above, the processing device 10 generates the glass coordinate system LCB based on the sensor data acquired from the AR glass 20B. If the AR glass 20B is used for a long time, an error occurs in the sensor data of the AR glass 20B. Therefore, the region where the glass content GC and the PJ content PJC are superimposed is deviated from the region OVL which should be superimposed. The processing device 10 corrects the PJ coordinate system LCC in order to suppress the deviation. By this correction, the relative position between the glass content GC and the PJ content PJC is corrected.

9 and 10 are diagrams showing a method of correcting the relative position between the glass content GC and the PJ content PJC. FIG. 9 is a diagram showing a method adopted in this embodiment. FIG. 10 is a diagram showing a comparative example.

As shown in FIG. 9, when the finger of the second user U2 is superimposed on the glass content GC, the gesture detection unit 14 recognizes the finger as the operation site OP. The gesture detection unit 14 determines that the operation has been started by the second user U2. The correction unit 17 corrects the glass coordinate system LCA and the PJ coordinate system LCC with reference to the glass coordinate system LCB. As a result, the position of the PJ content PJC displayed by the projector 20C moves. The position of the glass content GC recognized by the second user U2 as an operation target does not change. Therefore, the second user U2 can continue to operate the glass content GC without any discomfort.

Although not shown, the position of the glass content GC displayed on the AR glass 20A also changes by correcting the relative position of the glass coordinate system LCA. Therefore, the position of the glass content GC recognized by the first user U1 via the AR glass 20A changes. However, the first user U1 is only looking at the operation of the second user U2, and does not perform an operation on the glass content GC. Therefore, even if the position of the glass content GC recognized by the first user U1 changes slightly, the first user U1 does not feel a great deal of discomfort.

In the comparative example of FIG. 10, when the second user U2 starts the operation of the glass content GC, the glass coordinate system LCA and the glass coordinate system LCB are corrected with reference to the PJ coordinate system LCC. The reason why the PJ coordinate system LCC is used as the correction reference is that the PJ coordinate system LCC is less likely to cause misalignment than the glass coordinate system LCB generated by using SLAM.

However, in the example of FIG. 10, the position of the PJ content PJC displayed by the projector 20C does not change, but the position of the glass content GC recognized by the second user U2 as an operation target changes. Since the position of the glass content GC to be operated changes when the second user U2 starts the operation, the second user U2 feels a great sense of discomfort. When the movement amount of the glass content GC is large, the finger is deviated from the glass content GC, and the operation for the glass content GC cannot be performed.

As described above, if the position of the content CT to be operated changes at the start of the operation, a great sense of discomfort occurs in the operating subject. Therefore, at the start of the operation, the relative position of the other local coordinate system LC is corrected with reference to the local coordinate system LC used for generating the content CT recognized as the operation target by the operation subject. After the operation of the content CT is completed, even if the position of the content CT to be operated changes, the operation subject does not feel a great discomfort. Therefore, after the operation of the content CT is completed, the glass coordinate system LCA and the glass coordinate system LCB are corrected with reference to the PJ coordinate system LCC. By using the PJ coordinate system LCC as a reference, the positional shift between the real object of the real space RS and the content CT of the AR glasses 20A and the AR glasses 20B is less likely to occur.

11 to 13 are diagrams showing other application examples of the information processing system 1.

The processing device 10 combines the PJ content PJC and the glass content GC to form one slider. PJ Content PJC displays a slider track. The glass content GC displays a slider knob and a part of the track. The AR glass 20A and the AR glass 20B set the glass content area GCA at a position superimposing on the PJ content PJC, and display the glass content GC in the glass content area GCA.

In FIG. 11, the PJ content PJC and the glass content GC are displayed without causing misalignment with each other. In FIG. 12, there is a misalignment between the PJ content PJC and the glass content GC. The glass content GC is displayed at a position closer to the viewpoint of the user U than the PJ content PJC. Therefore, the area SHA of the PJ content PJC shielded by the glass content area GCA is widened. Further, a part of the tracks included in the glass content GC intersects with the tracks displayed by the PJ content PJC. As shown in FIG. 13, when the user U starts to operate the knob of the slider, the position of the PJ content PJC is corrected according to the position of the glass content GC.

In FIG. 13, the relative position of the PJ content PJC is corrected so that a part of the track included in the glass content GC and the track displayed by the PJ content PJC are parallel to each other. However, the correction method is not limited to this. The following correction methods are also conceivable.

(A) Correction at one point The correction unit 17 corrects the local coordinate system LC by a coordinate transformation process for translating the local coordinate system LC. When the coordinates of one operation point are detected as the coordinates of the operation site OP by a touch operation or the like, the correction unit 17 obtains the difference between the coordinates of the two local coordinate systems LC indicating the same operation point. The correction unit 17 translates one local coordinate system LC based on the difference.

(B) Correction at two points The correction unit 17 corrects the local coordinate system LC by translation processing that translates the local coordinate system LC and rotates the local coordinate system LC around one rotation axis. conduct. When the coordinates of the two operation points are detected as the coordinates of the operation site OP by the operation of the slider or the like, the correction unit 17 obtains a straight line passing through the two operation points in each of the two local coordinate system LCs. The correction unit 17 translates and rotates one local coordinate system LC so that the two straight lines obtained by the two local coordinate system LCs match.

(C) Correction at 3 points The correction unit 17 corrects the local coordinate system LC by translation processing that translates the local coordinate system LC and rotates the local coordinate system LC around the two rotation axes. conduct. When the coordinates of three operation points that are not on the same straight line are detected as the coordinates of the operation site OP due to the movement of the content CT or the like, the correction unit 17 performs the three operation points in each of the two local coordinate system LCs. Find the plane containing. The correction unit 17 translates and rotates one local coordinate system LC so that the two planes obtained by the two local coordinate system LCs match.

[1-2. Information processing method]
FIG. 14 is a flowchart showing the information processing method of the present embodiment.

In step S1, the gesture detection unit 14 determines whether or not the content CT operation has been performed by the user U. If it is determined that the content CT operation has been performed in step S1 (step S1: Yes), the process proceeds to step S2. If it is determined in step S1 that the content CT operation has not been performed (step S1: No), step S1 is repeated until the content CT operation is detected.

In step S2, the error information detection unit 16 determines whether or not an error has occurred in the relative positions of the plurality of local coordinate system LCs. If it is determined that an error has occurred in step S2 (step S2: Yes), the process proceeds to step S3. If it is determined in step S2 that no error has occurred (step S2: No), the process ends.

In step S3, the gesture detection unit 14 determines whether or not the target of the operation by the user U is the PJ content PJC. If it is determined in step S3 that the operation target is PJ content PJC (step S3: Yes), the process proceeds to step S4. If it is determined in step S3 that the operation target is not the PJ content PJC (step S3: No), the process proceeds to step S8.

In step S4, the interaction detection unit 15 determines the accuracy of the sensor data output from the PJ sensor data acquisition unit 12. If the accuracy of the sensor data is larger than the threshold value in step S4 (step S4: No), the process proceeds to step S5. If the accuracy of the sensor data is equal to or less than the threshold value in step S4 (step S4: Yes), the process proceeds to step S16. In step S16, a correction error is output and correction is not performed.

In step S5, the correction unit 17 corrects the relative positions of the glass coordinate system LCA and the glass coordinate system LCB with reference to the PJ coordinate system LCC. In step S6, the correction unit 17 recalculates the positions of the glass contents GC of the AR glasses 20A and the AR glasses 20B by using the first coordinate conversion formula and the second coordinate conversion formula. In step S7, the display control unit 19 corrects the displayed position of the glass content GC based on the position calculated by the correction unit 17.

In step S8, the gesture detection unit 14 determines whether or not the target of the operation by the user U is the glass content GC. If it is determined in step S8 that the operation target is the glass content GC (step S8: Yes), the process proceeds to step S9. If it is determined in step S8 that the operation target is not the glass content GC (step S8: No), the process proceeds to step S16. In step S16, a correction error is output and correction is not performed.

In step S9, the correction unit 17 calculates the position where the position of the PJ content PJC is temporarily corrected. Here, first, the correction unit 17 corrects the relative position of the PJ coordinate system LCC with reference to the glass coordinate system used to generate the glass content GC recognized as the operation target by the operation subject. Then, the correction unit 17 calculates the position of the PJ content PJC by using a coordinate conversion formula that performs coordinate conversion between the glass coordinate system and the PJ coordinate system LCC.

In step S10, the display control unit 19 corrects the displayed position of the PJ content PJC based on the position calculated by the correction unit 17.

In step S11, the gesture detection unit 14 determines whether or not the operation of the glass content GC by the user U is completed. If it is determined in step S11 that the operation of the glass content GC is completed (step S11: Yes), the process proceeds to step S12. If it is not determined in step S11 that the operation of the glass content GC is completed (step S11: No), the process returns to step S9.

In step S12, the correction unit 17 corrects the relative positions of the glass coordinate system LCA and the glass coordinate system LCB with reference to the PJ coordinate system LCC. In step S13, the correction unit 17 recalculates the positions of the AR glass 20A and the AR glass 20B glass content GC by using the first coordinate conversion formula and the second coordinate conversion formula. In step S14, the display control unit 19 corrects the displayed position of the glass content GC based on the position calculated by the correction unit 17.

In step S15, the correction unit 17 ends the correction of the PJ coordinate system LCC with reference to the glass coordinate system. The display control unit 19 moves the position of the PJ content PJC to the position set by using the PJ coordinate system LCC.

[1-3. effect]
The information processing system 1 has an error information detection unit 16 and a correction unit 17. The error information detection unit 16 detects information on errors that occur at relative positions of a plurality of local coordinate systems LC. In response to the start of the operation of the content CT, the correction unit 17 corrects the relative positions of all the other local coordinate system LCs with respect to the specific local coordinate system LC based on the error information. The specific local coordinate system LC is a local coordinate system LC used to generate a content CT recognized as an operation target by the operating subject of the content CT among a plurality of local coordinate system LCs. In the information processing method of the present embodiment, the information processing of the above-mentioned information processing system 1 is executed by a computer. The program 31 of the present embodiment makes the computer realize the information processing of the above-mentioned information processing system 1.

According to this configuration, at the start of the operation of the content CT, the local coordinate system LC (origin position and coordinate axis position) other than the specific local coordinate system LC has an error in the relative position with the specific local coordinate system LC. Move to no position. The local coordinate system LC (specific local coordinate system LC) related to the generation of the content CT recognized by the operating subject does not move. Therefore, the position of the content CT recognized by the operating subject is not changed. Therefore, the operating subject can continue to operate the content CT without feeling uncomfortable.

The error information detection unit 16 detects error information based on, for example, the coordinates of the part OP that interacts with the content CT of the operation subject, which is detected for each local coordinate system LC.

According to this configuration, the position of the operation site OP is detected using the coordinates of each local coordinate system LC. By comparing the coordinates of the operation site OP in each local coordinate system LC, an error generated in the relative position between the local coordinate system LCs is detected. In this configuration, the above error is detected at the timing when the operation is performed. Therefore, the local coordinate system LC can be corrected at the timing when the operation is started.

Further, in this configuration, a member called a marker for correcting the position / orientation information of the device 20 is unnecessary. Markers are used in fields such as AR glasses and head-mounted displays. The marker is installed at a specific position in the real space RS. The user U equipped with the device 20 moves to the position of the marker when correcting the error generated in the position / attitude information, and takes a picture of the marker with the camera. By comparing the image of the marker stored in the device 20 with the image of the marker taken by the camera, an error in the position / orientation information (error generated in the relative position between the world coordinate system WC and the local coordinate system LC) can be obtained. It can be corrected.

In the correction method using the marker, the user U must move to the position of the marker. However, in the present embodiment, by comparing the coordinates of the operation site OP detected for each device 20, an error generated in the relative position between the local coordinate system LCs is detected. Therefore, the local coordinate system LC can be easily corrected without using a marker.

For example, the correction unit 17 ends the correction based on the specific local coordinate system LC in response to the completion of the operation of the content CT. After that, the correction unit 17 corrects, for example, the relative positions of all other local coordinate system LCs with respect to the reference local coordinate system based on the error information. The reference local coordinate system is, for example, a local coordinate system LC having a smaller error in a position relative to the world coordinate system than a specific local coordinate system LC.

According to this configuration, the local coordinate system LC that is the reference for correction is switched from the specific local coordinate system LC to the reference local coordinate system at the timing when the operation of the content CT is completed. Since the coordinates are corrected with reference to the reference local coordinate system, the positional deviation between the real object of the real space RS and the content CT of each device 20 is unlikely to occur.

The information processing system 1 has, for example, a display control unit 19. For example, the display control unit 19 gradually changes the position of the content CT to the position calculated by the correction based on the reference local coordinate system in response to the completion of the operation of the content CT.

According to this configuration, the position of the content CT does not change suddenly. Therefore, the operation subject is less likely to feel uncomfortable.

The correction unit 17 corrects the local coordinate system LC by, for example, a coordinate conversion process for translating the local coordinate system LC.

According to this configuration, the correction of the local coordinate system LC is performed with a small amount of calculation.

The correction unit 17 corrects the local coordinate system LC by, for example, a coordinate transformation process for translating the local coordinate system LC and rotating the local coordinate system LC around one or more rotation axes.

According to this configuration, the correction of the local coordinate system LC is performed with high accuracy.

[2. Second Embodiment]
[2-1. Information processing system configuration]
FIG. 15 is a schematic diagram of the information processing system 2 of the second embodiment.
The difference from the first embodiment in this embodiment is that the content CT used is only the glass content GC, and the reference local coordinate system is determined based on the operation history of a plurality of devices. Hereinafter, the differences from the first embodiment will be mainly described.

FIG. 15 shows an example in which the information processing system 2 is applied to techno sports. A device 20 for displaying the content CT is attached to the heads of a plurality of users U who are players. As the device 20, for example, AR glass 20A is used. The AR glass 20A displays a virtual object showing the ball as the glass content GC. The user U puts the ball displayed as the glass content GC into the opponent's goal and competes for the number of goals.

The processing device 40 has, for example, a reference local coordinate system determination unit 41. The reference local coordinate system determination unit 41 determines, for example, the glass coordinate system having the smallest error in the relative position with the world coordinate system WC as the reference local coordinate system based on the operation history of the plurality of AR glasses 20A.

The operation history of the AR glass 20A is determined based on, for example, the sensor data detected by the AR glass 20A. For example, the space recognition unit 13 uses SLAM to create an environment map for each AR glass 20A and estimate the position of the AR glass 20A in the environment map. The reference local coordinate system determination unit 41 has been using the AR glass 20A since the last time the relative position of the glass coordinate system LCA was corrected with respect to the reference local coordinate system based on the movement history of the AR glass 20A in the environment map. Calculates the distance traveled by. The reference local coordinate system determination unit 41 compares the movement distances after the correction processing using the reference local coordinate system calculated for each AR glass 20A, and uses the glass coordinate system LCA of the AR glass 20A having the smallest movement distance as a reference. Judged as a local coordinate system.

The reference local coordinate system determination unit 41 may determine the reference local coordinate system based on the history of acceleration for each AR glass 20A. For example, the reference local coordinate system determination unit 41 detects the acceleration history for each AR glass 20A based on the acceleration data included in the sensor data. Based on the acceleration history of the AR glass 20A, the reference local coordinate system determination unit 41 has AR that exceeds the threshold value since the last correction of the relative position of the glass coordinate system with respect to the reference local coordinate system. The time applied to the glass 20A is calculated. The reference local coordinate system determination unit 41 compares the above times calculated for each AR glass 20A, and determines that the glass coordinate system LCA of the AR glass 20A having the shortest time is the reference local coordinate system. The threshold value is included in the registration data 52, for example.

The correction unit 17 corrects the relative positions of all other glass coordinate system LCA with respect to the specific glass coordinate system LCA based on the error information in response to the start of the operation of the glass content GC. The specific glass coordinate system LCA is the glass coordinate system LCA used to generate the glass content GC recognized as the operation target by the operating subject of the glass content GC. By this correction, for example, the origin position and the position of the coordinate axis of other glass coordinate system LCA other than the specific glass coordinate system LCA move to a position where the error of the relative position with the specific glass coordinate system LCA becomes small.

The correction unit 17 ends the correction based on the specific glass coordinate system LCA in response to the completion of the operation of the glass content GC. Then, the correction unit 17 switches, for example, the glass coordinate system LCA that is the reference for correction from the specific glass coordinate system LCA to the reference local coordinate system determined by the reference local coordinate system determination unit 41. The correction unit 17 corrects, for example, the relative positions of all other glass coordinate system LCA with respect to the reference local coordinate system based on the error information. By this correction, for example, the origin position and the position of the coordinate axis of the glass coordinate system LCA other than the reference local coordinate system are moved to the position where the error of the relative position with the reference local coordinate system becomes small.

The storage device 50 stores, for example, the program 51 executed by the processing device 40 and the registration data 52. The program 51 is a program for causing a computer to execute information processing according to the present embodiment. The processing device 40 performs various processes according to the program 51 stored in the storage device 50. By executing the program 51, the processing device 40 executes the glass sensor data acquisition unit 11, the space recognition unit 13, the gesture detection unit 14, the reference local coordinate system determination unit 41, the error information detection unit 16, the correction unit 17, and the content generation. It functions as a unit 18 and a display control unit 19.

[2-2. effect]
In this embodiment, the reference local coordinate system is determined based on the operation history of the plurality of AR glasses 20A. Therefore, even when the local coordinate system LC is deviated due to the operation of the AR glass 20A, the reference local coordinate system as the reference for correction can be appropriately selected.

[2-3. Modification example]
16 and 17 are diagrams showing an example in which the information processing system 2 is applied to the correction of the local coordinate system LC set in the robot RB.

As shown in FIG. 16, the plurality of devices 20 to be processed include the AR glass 20A and the robot RB. The robot RB is, for example, a robot that autonomously moves using SLAM. The robot coordinate system LCR is set as the local coordinate system LC in the robot RB.

For example, the processing device 40 uses the robot coordinate system LCR to generate a content CT (robot content RC) representing a ball. The robot RB recognizes that the ball exists in the real space RS. The AR glass 20A superimposes and displays the glass content GC representing the same ball on the real space RS.

Also in this modification, the relative of the local coordinate system LC other than the specific local coordinate system LC used to generate the content CT recognized by the operating subject when the robot RB or the user U starts the operation on the content CT. The position is corrected. Then, after the operation to the content CT is completed, the local coordinate system LC that is the reference of the correction is switched from the specific local coordinate system LC to the reference local coordinate system. The correction method of the local coordinate system LC is the same as that described in the second embodiment.

For example, the robot RB has a plurality of sensors including a depth sensor, an acceleration sensor, a microphone and a camera. The robot RB recognizes the real space RS around the robot RB, estimates the position of the robot RB in the real space RS, and the real object around the robot RB based on the sensor data detected by a plurality of sensors. Perform detection. Recognition of the real space RS and estimation of the position in the real space RS are performed using, for example, SLAM. The robot RB outputs, for example, the robot coordinate system LCR information (information regarding the origin position and the position of the coordinate axis) and the environment map information generated by using SLAM to the error information detection unit 16.

As shown in FIG. 17, the robot RB detects the coordinates of the operation part OP (the part where the robot RB interacts with the robot content RC) in the robot coordinate system LCR based on the sensor data. The robot RB outputs information on the coordinates of the operation site OP to the error information detection unit 16. The error information detection unit 16 is, for example, a relative position between the robot coordinate system LCR and the glass coordinate system LCA based on the coordinates of the operation site OP in the robot coordinate system LCR and the coordinates of the operation site OP in the glass coordinate system LCA. Detects the information of the error that occurred in.

For example, the correction unit 17 corrects the relative position of the glass coordinate system LCA with respect to the robot coordinate system LCR based on the error information in response to the start of the operation of the content CT. By this correction, the position of the glass content GC recognized by the user U via the AR glass 20A is corrected according to the position of the robot content RC recognized by the robot RB as an operation target.

The reference local coordinate system determination unit 41 determines the reference local coordinate system based on the operation history of the robot RB and the operation history of the AR glass 20A. When the reference local coordinate system is the glass coordinate system LCA, the correction unit 17 sets the local coordinate system LC as the correction reference from the robot coordinate system LCR to the glass when the operation of the robot content RC by the robot RB is completed. Switch to the coordinate system LCA.

In this modification, when the robot RB operates the robot content RC, the position of the robot content RC to be operated does not change. Therefore, the operation of the robot RB is less likely to be confused.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

Note that this technology can also take the following configurations.

(1)
An error information detector that detects information on errors that occur at relative positions in multiple local coordinate systems,
All other local coordinate systems for the particular local coordinate system used to generate the content recognized by the content operator among the plurality of local coordinate systems in response to the initiation of the content operation. A correction unit that corrects the relative position of the
Information processing system with.
(2)
The information processing system according to (1) above, wherein the error information detection unit detects information on the error based on the coordinates of a portion that interacts with the content of the operating subject, which is detected for each local coordinate system.
(3)
In response to the completion of the operation of the content, the correction unit ends the correction based on the specific local coordinate system, and occurs at a position relative to the world coordinate system rather than the specific local coordinate system. The information processing system according to (1) or (2) above, which corrects the relative positions of all other local coordinate systems with respect to the reference local coordinate system having a small error based on the information of the error.
(4)
Described in (3) above, the present invention has a display control unit having a display control unit that gradually changes the position of the content to a position calculated by a correction based on the reference local coordinate system in response to the completion of the operation of the content. Information processing system.
(5)
Based on the operation history of a plurality of devices in which the plurality of local coordinate systems are set, the local coordinate system having the smallest error in the position relative to the world coordinate system is determined to be the reference local coordinate system. The information processing system according to (3) or (4) above, which has a coordinate system determination unit.
(6)
The information processing system according to any one of (1) to (5) above, wherein the correction unit corrects the local coordinate system by a coordinate transformation process for translating the local coordinate system.
(7)
The correction unit corrects the local coordinate system by translation processing for parallel movement of the local coordinate system and rotational movement of the local coordinate system around one or more rotation axes (1) to (5). The information processing system according to any one of the above.
(8)
Detects information on errors that occur in relative positions in multiple local coordinate systems,
All other local coordinate systems for the particular local coordinate system used to generate the content recognized by the content operator among the plurality of local coordinate systems in response to the initiation of the content operation. The relative position of is corrected based on the above error information.
A method of information processing performed by a computer that has.
(9)
Detects information on errors that occur in relative positions in multiple local coordinate systems,
All other local coordinate systems for the particular local coordinate system used to generate the content recognized by the content operator among the plurality of local coordinate systems in response to the initiation of the content operation. The relative position of is corrected based on the above error information.
A program that makes a computer realize that.

1, 2, Information processing system 16 Error information detection unit 17 Correction unit 19 Display control unit 41 Reference local coordinate system judgment unit CT Content LC Local coordinate system LCC PJ coordinate system (reference local coordinate system)
OP operation site

Claims

An error information detector that detects information on errors that occur at relative positions in multiple local coordinate systems,
All other local coordinate systems used to generate the content that the operator of the content recognizes as an operation target in response to the initiation of the operation of the content. A correction unit that corrects the relative position of the local coordinate system based on the error information,
Information processing system with.
The information processing system according to claim 1, wherein the error information detection unit detects the error information based on the coordinates of the portion that interacts with the content of the operating subject, which is detected for each local coordinate system.
In response to the completion of the operation of the content, the correction unit ends the correction based on the specific local coordinate system, and occurs at a position relative to the world coordinate system rather than the specific local coordinate system. The information processing system according to claim 1, wherein the relative positions of all other local coordinate systems with respect to the reference local coordinate system having a small error are corrected based on the information of the error.
The third aspect of claim 3 has a display control unit having a display control unit that gradually changes the position of the content to a position calculated by a correction based on the reference local coordinate system in response to the completion of the operation of the content. Information processing system.
Based on the operation history of a plurality of devices in which the plurality of local coordinate systems are set, the local coordinate system having the smallest error in the position relative to the world coordinate system is determined to be the reference local coordinate system. The information processing system according to claim 3, which has a coordinate system determination unit.
The information processing system according to claim 1, wherein the correction unit corrects the local coordinate system by a coordinate transformation process for translating the local coordinate system.
The information processing according to claim 1, wherein the correction unit corrects the local coordinate system by translation processing for parallel movement of the local coordinate system and rotational movement of the local coordinate system around one or more rotation axes. system.
Detects information on errors that occur in relative positions in multiple local coordinate systems,
All other local coordinate systems used to generate the content that the operator of the content recognizes as an operation target in response to the initiation of the operation of the content. Correct the relative position of the local coordinate system based on the error information,
A method of information processing performed by a computer that has.
Detects information on errors that occur in relative positions in multiple local coordinate systems,
In response to the initiation of the operation of the content, all the other local coordinate systems used to generate the content that the operating subject of the content recognizes as the operation target among the plurality of local coordinate systems. Correct the relative position of the local coordinate system based on the error information,
A program that makes a computer realize that.