JP7106097B2 - telepresence system - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law Posted on December 1, 2017, URL: https://www. csie. ntu. edu. tw/~tai 2017

The present invention relates to telepresence systems.

2. Description of the Related Art In recent years, video conferencing systems are being used when a person working at home has a conference with an employee at a company, or when a conference is held between distant offices. A telepresence system is known as a teleconferencing system in which a robot gives a sense of realism such that the user can feel as if he or she is actually there even though the user is in a remote location. Telepresence systems are also known for communicating between family members living apart.
Regarding the telepresence system, the following techniques are conventionally known.

Non-Patent Document 1 describes a telepresence system in which a trolley with wheels is equipped with a monitor that displays the other party's face, a camera that captures itself, a microphone (small speaker and microphone), a robot arm, a laser pointer, etc. A robot is described. With the technology described in Non-Patent Document 1, it is possible to have a conversation with the other party projected on the telepresence robot, perform simple gestures with the robot arm, and point to an object with a laser pointer. there is That is, communication through sight and hearing is possible.

In Non-Patent Document 2, a subject manipulates a robot to move back and forth and rotate, so that it feels as if he or she can actually talk to and be seen by a remote partner in the same room. A telepresence robot intended to be a telepresence robot is described.
Non-Patent Literature 3 describes a technique for detecting a nod or backtracking of a remote participant and performing an exaggerated expression using a telepresence robot.

Annica Kristoffersson, et.al, “A Revier of Mobile Robotic Telepresence,” Advances in Human-Computer Interaction, Vol. 2013, Article ID 902316, 17 pages, 2013. doi:10. 1155/2013/902316. Yuki Murakami and 3 others, “Effect of movement of robot-mounted camera on telepresence,” Transactions of Information Processing Society of Japan, 1882-7764, 2010-01-15, 51, 1, 54-62, URL<https://ci. nii.ac.jp/naid/110007970615/> Komei Hasegawa and one other person, “Proposal of Remote Conversation Robot that Exaggerates Gestures to Increase Opportunities for Remote Participants to Speak (Cloud Network Robot)” The Institute of Electronics, Information and Communication Engineers Technical Report =IIEICE technical report, IEICE Technical Report, 0913- 5685, The Institute of Electronics, Information and Communication Engineers, 2014-09-26, 114, 227, 5-10, URL<https://ci.nii.ac.jp/naid/110009950799/>

(Problem of conventional technology)
In the prior art described in Non-Patent Document 1, simple gestures can be transmitted to the other party, but, for example, actions during conversation are not automatically transmitted to the other party. In the prior art described in Non-Patent Document 2, the subject's maneuvering result is also transmitted to the other party, but the motion of the subject during conversation is not automatically transmitted to the other party. In the techniques described in Non-Patent Documents 1 and 2, for example, nodding motions (vertical shaking motions of the head) indicating sympathy or agreement, etc., and horizontal shaking motions indicating denial or disapproval, etc. during a conversation are automatically performed. is not transmitted physically. In particular, in remote dialogue, when a nodding motion or the like is transmitted, it is easy to communicate feelings and intentions, and communication is easy to progress.

The technology described in Non-Patent Literature 3, in which a nod or the like is detected and the telepresence robot of the other party expresses itself, facilitates communication of emotions and intentions. Here, in the telepresence robot, when the monitor (display) that displays the other party's face is different from the head of the robot that expresses nodding, that is, the monitor is the chest of the robot as in Non-Patent Document 2. When the robot is installed in a remote location, even if the head of the robot nods while looking at the other person on the monitor, the person may not notice the nod, making communication difficult. Therefore, it is preferable to vibrate the monitor displaying the image of the other party's face vertically in response to the nodding motion of the other party, or to vibrate left and right in accordance with the manifestation of intention such as denial. However, when the monitor is moved in accordance with the movement of the user's head, the monitor moves frequently, and the user who is looking at the monitor may feel stress or suffer from motion sickness. There is

SUMMARY OF THE INVENTION It is a technical object of the present invention to reduce the user's stress while facilitating communication when making a remote call using a telepresence system.

In order to solve the technical problem, the telepresence system of the invention described in claim 1 is:
a first imaging unit that acquires an image of the first user; a first audio input unit that acquires the audio of the first user; a first display unit that displays the image; a first terminal having a first slope that slopes the display;
A second terminal that is installed at a location separated from the first terminal and that is capable of transmitting and receiving information to and from the first terminal, and that acquires an image of a second user. a shooting unit, a second audio input unit that acquires the voice of the second user, a second display unit that displays an image, and a second tilt unit that tilts the second display unit a second terminal having
Action discrimination means for discriminating respective actions of the first user and the second user based on images taken by the first imaging unit and the second imaging unit ;
One user's voice is input to one of the first terminal and the second terminal, and the other user performs a predetermined operation on the other of the first terminal and the second terminal is performed, the inclined portion of one of the first terminal and the second terminal is inclined according to the motion of the other user, and the first terminal and the second terminal tilt control means for not operating the tilt portion of the other terminal of the
characterized by comprising

The invention according to claim 2 is the telepresence system according to claim 1,
said tilt control means for tilting one of said first terminal and said second terminal in a vertical direction when the other user of said first terminal and said second terminal performs a nodding motion; characterized by comprising

The invention according to claim 3 is the telepresence system according to claim 1 or 2,
When the other terminal of the first terminal and the second terminal acquires a voice that does not reach a predetermined time, the input of the voice is ignored.

According to the first aspect of the invention, when one user's voice is input and the other user performs a predetermined action on the other side, the inclined portion of the one terminal is By tilting according to the movement of the other user and not moving the tilting part of the other terminal, when making a remote call using the telepresence system, the user's stress can be reduced while smooth communication. can be mitigated.
According to the second aspect of the invention, the nodding action of the listener of the conversation can be conveyed to the speaker.
According to the third aspect of the present invention, it is possible to reduce the user's stress compared to the case where the inclined portion of the other terminal operates even when a short speech is made.

FIG. 1 is an overall explanatory diagram of a telepresence system according to Embodiment 1 of the present invention. FIG. 2 is an explanatory diagram of the telepresence robot of the first embodiment. FIG. 3 is a functional block diagram of a telepresence robot in the remote communication system of embodiment 1. FIG.

Next, specific examples of embodiments of the present invention (hereinafter referred to as examples) will be described with reference to the drawings, but the present invention is not limited to the following examples.
It should be noted that in the following explanation using the drawings, illustration of members other than those necessary for the explanation is omitted as appropriate for ease of understanding.

FIG. 1 is an overall explanatory diagram of a telepresence system according to Embodiment 1 of the present invention.
In FIG. 1, a remote communication system S as an example of the telepresence system of the present invention has telepresence robots 1 and 2 as an example of terminals that can be used by users. Telepresence robots 1 and 2 are installed at installation locations 3 and 4 apart from each other. For example, a first telepresence robot (first terminal) 1 is installed in a grandchild's house (first installation location) 3 as an example of a user, and a second telepresence robot 2 (second terminal ) can be installed in a grandparent's house (second installation location) 4 as an example of the user.

FIG. 2 is an explanatory diagram of the telepresence robot of the first embodiment.
1 and 2, the first telepresence robot 1 has a pedestal 11. As shown in FIG. A column portion 12 extending upward is supported on the base portion 11 . An oscillating head (first inclined portion) 13 that can move up and down (tilt) and left and right (pan) is supported on the upper end portion of the column portion 12 . The oscillating head 13 has a conventionally known structure that can be operated by being driven by a drive source (not shown). A tablet 14 as an example of a terminal body is supported by the swing head 13 . The tablet 14 is an example of a first display unit and has a touch panel 14a as an example of an input unit. The touch panel 14a is configured to be capable of displaying an image and allowing input by being touched with an input means such as a finger or a pen.

Further, the tablet 14 is provided with a microphone 14b as an example of a first voice input section. The microphone 14b can pick up sounds around the tablet 14, such as the user's voice. Furthermore, the tablet 14 is also provided with a speaker (not shown) as an example of a first audio output unit.
Further, in the first telepresence robot 1 of Example 1, a camera 16 as an example of a first imaging unit is installed on the pillar 12 . The camera 16 can acquire a user's image or the like.

Since the second telepresence robot 2 is configured in the same manner as the first telepresence robot 1, detailed description thereof will be omitted. That is, the second telepresence robot 2 includes an oscillating head as an example of the second inclined section, a touch panel as an example of the second display section, a microphone as an example of the second voice input section, and a second voice input section as an example. and a speaker as an example of the second audio output unit.
Each of the telepresence robots 1 and 2 is connected so as to be able to communicate information via an Internet line 21 as an example of a public line. Any connection method can be adopted for the Internet line 21, whether it is a wired connection or a wireless connection.

(Description of the control unit of the telepresence robot 1 of the first embodiment)
FIG. 3 is a functional block diagram of a telepresence robot in the remote communication system of embodiment 1. FIG.
In FIG. 3, the control unit 31 of the telepresence robots 1 and 2 of the first embodiment includes an I/O (input/output interface) for inputting/outputting signals with the outside, adjusting the input/output signal level, etc., necessary startup processing, and the like. ROM (read-only memory) that stores programs and data, etc. for performing, RAM (random access memory) for temporarily storing necessary data and programs, boot program stored in ROM etc. It is composed of a computer device having a CPU (central processing unit) that performs processing and a clock oscillator, etc. Various functions can be realized by executing programs stored in the ROM and RAM.
The control unit 31 of the telepresence robots 1 and 2 stores basic software for controlling basic operations, a so-called operating system OS, a communication program as an example of an application program, and other software (not shown).
In the following description, the first telepresence robot 1 will be described, but the second telepresence robot 2 is the same as the first telepresence robot 1, so detailed description will be omitted.

(Elements connected to the control unit 31 of the first embodiment)
The control unit 31 of the first telepresence robot 1 receives output signals from signal output elements such as the touch panel 14a, the microphone 14b, the camera 16, and the like.
Further, the control unit 31 of the first embodiment outputs control signals to controlled elements such as the touch panel 14a, the swing head 13, and the speaker.

(Function of first telepresence robot 1)
The communication program of the control unit 31 of the first telepresence robot 1 of Example 1 has the following functional units (program modules).
A photographing means 41 photographs an image via the camera 16 . The video imaged by the imaging means 41 is transmitted to the second telepresence robot 2 via the Internet line 21 .
The voice acquisition means 42 acquires voice through the microphone 14b. The voice acquired by the voice acquisition means 42 is transmitted to the second telepresence robot 2 via the Internet line 21 .

The motion discrimination means 43 discriminates the motion of the user based on the image captured by the camera 16 . The motion determining means 43 of the first embodiment performs image analysis from the image captured by the camera 16 to identify the user's head (detection of eyes, nose, and mouth), and determine the motion of the head (shaking the head vertically ( nodding), shaking the head sideways, tilting the head, etc.).
In the first embodiment, the determination result of the action determination means 43 is used by the first telepresence robot 1 and is also transmitted to the second telepresence robot 2, but is not limited to this. For example, it is possible to determine the action of the other party from the image transmitted from the telepresence robot 2 of the other party without transmitting to the telepresence robot 2 of the other party.

The utterance discriminating means 44 discriminates the utterance of the user based on the voice acquired by the microphone 14b. The utterance determination unit 44 of the first embodiment determines that no utterance is made when a voice is acquired that does not reach a predetermined time. In Example 1, as an example, the predetermined time is set to 2 seconds. Therefore, short utterances such as "yes", "heh~", "no", etc. that are used to express agreement, sympathy, or denial during conversation are not identified as utterances in the user's conversation. Speech that lasts longer than 2 seconds is determined to be uttered by the user in order to converse with the other party. It should be noted that the utterance discrimination means 44 can also be configured to discriminate whether or not the other party has spoken, based on the voice information transmitted from the other party, similarly to the action discrimination means 43 .

The image display means 45 displays the image captured by the image capturing means 41 of the second telepresence robot 2 and received via the Internet line 21 on the touch panel 14a.
The voice output means 46 outputs the voice acquired by the voice acquisition means 42 of the second telepresence robot 2 and received via the Internet line 21 from a speaker. Note that the voice output unit 46 of the first embodiment also outputs short-time speech that the speech discrimination unit 44 did not discriminate as speech from the speaker.

The tilt control means 47 controls the swing head 13 to tilt the tablet 14 . In the tilt control means 47 of the first embodiment, the voice of the user (self) at the installation location 3 is input in the first telepresence robot 1, and the voice of the user (self) at the installation location 4 is input in the second telepresence robot 2. When the other party performs a predetermined action (such as nodding), the swing head 13 of the first telepresence robot 1 is tilted according to the action of the other party. Further, the tilt control means 47 of the first embodiment controls the second telepresence robot 2 when the other party's voice is input to the second telepresence robot 2 and when the first telepresence robot 1 is not speaking. Even if the telepresence robot 2 performs a predetermined action (such as nodding), the swinging head 13 of the first telepresence robot 1 is not operated. That is, the oscillating head 13 of the speaking side (speaker) operates according to the movement of the other party (listener), but the oscillating head 13 of the listener does not operate (even if the speaker nods). When both of them are speaking (both are speakers), both of the swinging heads 13 operate according to the movement of the other's head.

The tilt control means 47 of the first embodiment controls the tilting head 13 according to the motion determined by the motion determining means 43, such as a motion of shaking the head vertically (nodding), a motion of shaking the head sideways, or a motion of tilting the head. to move the tablet 14 vertically, horizontally, or tilt it. Note that the operation of the swinging head 13 can be performed according to the amount of movement of the opponent's head (inclination angle, movement speed, etc.), but is not limited to this. For example, when the motion determining means 43 of the second telepresence robot 2 determines that it is a "nod" and a signal indicating the "nod" motion is input to the first telepresence robot 1, the "tilt angle of 5°" is determined. It is also possible to configure the oscillating head 13 to perform a predetermined motion in response to a predetermined motion of the user, such as "moving up and down twice".

Further, the tilt control means 47 of the first embodiment does not operate the swinging head 13 when the speech discrimination means 44 does not discriminate speech. That is, even if a short utterance such as "Yes" is detected as described above, it is not determined as a speaker, but as a listener, and the swing head 13 does not operate. Therefore, in the first embodiment, short-time speech is formally ignored for the operation of the swinging head 13 .

(Action of Example 1)
In the remote communication system S of the first embodiment having the above configuration, when the grandfather and the grandchild who are in distant places communicate with each other, their faces are displayed on the tablets 14 of their telepresence robots 1, and their voices are displayed. communication is possible. In other words, as in the past, visual and auditory communication is possible.
When the listener performs an action such as nodding, the oscillating head 13 on the speaker side is activated. Therefore, the speaker side can visually recognize agreement, sympathy, doubt, disapproval, etc. of the other party by moving the tablet 14 when the other party nods or tilts the head. Therefore, communication can be achieved as compared with the case where the tablet 14 is not tilted.

On the other hand, on the listener's side, the swing head 13 does not operate. Since the listener does not speak, he often listens to the speaker's speech while looking at the other party (speaker) on the tablet 14 . At this time, if the tablet 14 moves frequently in accordance with the movement of the speaker, the screen that the listener pays attention to moves. Therefore, listeners may experience stress, discomfort, or motion sickness-like situations. On the other hand, in the first embodiment, the oscillating heads 13 of the telepresence robots 1 and 2 on the listening side do not operate. Therefore, even if the listener is paying attention to the tablet 14, the listener's stress is reduced.

Further, in the remote communication system S of the first embodiment, when a short speech is made, it is not determined that the listener has changed to the speaker, and the swinging head 13 does not operate. Therefore, when the listener utters an utterance such as "Yeah", "Yeah", or "Heh~", the listener's stress caused by the movement of the swinging head 13 can be reduced.

(Change example)
Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention described in the scope of claims. It is possible. Modified examples (H01) to (H08) of the present invention are exemplified below.
(H01) In the above embodiment, the configuration in which each of the telepresence robots 1 and 2 is provided with the control unit 31 was exemplified, but the present invention is not limited to this. A control unit is installed in a server connected to the Internet line 21, and the telepresence robots 1 and 2 are managed and controlled by the server. As a machine, it is also possible to adopt a configuration in which the parent machine manages and controls the child machine. In addition, it is also possible to adopt a configuration in which distributed processing is performed by a plurality of information processing terminals connected to the Internet line 21 .

(H02) In the above-described embodiment, the remote communication system S has two telepresence robots 1 and 2, but the configuration is not limited to this. Systems with more than two telepresence robots are also possible.
(H03) In the above embodiment, three motions of nodding (vertical swing of the neck), lateral swing of the neck, and tilting of the neck (inclination of the neck) are illustrated as examples of motions, but the present invention is not limited to these. For example, depending on the required functions and specifications, it is possible to adopt a configuration that distinguishes only nodding, or conversely, it is also possible to adopt a configuration that distinguishes four or more motions.

(H04) In the above embodiment, the oscillating head 13 has been exemplified as being capable of tilting (rotating) up, down, left and right, but it is not limited to this. For example, in a telepresence robot that only nods, it is possible to employ a swinging head 13 that can only move up and down, or to transmit complex head movements, depending on the required functions and specifications. is functionally required, it is possible to employ a high-performance oscillating head.

(H05) In the above embodiment, the configuration in which the camera 16 is installed on the pillar 12 was illustrated, but the configuration is not limited to this. For example, it is also possible to use a camera provided on the tablet 14 . However, in this case, when the listener nods, the speaker's tablet 14 is activated, the camera 16 of the tablet is also moved integrally, and the captured image is moved. This is not very preferable because the image displayed on the screen will fluctuate.

(H06) In the above embodiment, a grandfather and a grandchild were exemplified as an example of users, but the present invention is not limited to this. For example, meetings between offices, communication between hospitalized patients and their families at home, communication between elderly people living alone and their caregivers and family members, communication between elderly people in private rooms in nursing homes and their caregivers and family members, etc. It can be applied in any place of communication such as communication between an owner at a travel destination and a pet at home.
(H07) In the above-described embodiment, each of the telepresence robots 1 and 2 determines actions such as nodding and the presence/absence of short-term speech, transmits the results to the other party, and controls the head swing. Although the configuration is exemplified, it is not limited to this. It is also possible to configure each terminal to perform discrimination based on video and audio transmitted from the other party without transmitting the discrimination result.

(H08) In the above embodiment, the configuration using the tablet 14 having the display unit and the voice input unit as the terminal body was illustrated, but the configuration is not limited to this. Instead of using a commercially available tablet, it is possible to arbitrarily change such as installing a display panel supported by a swinging head or installing a microphone on the pillar 12 .

1... the first terminal,
2 ... second terminal,
13 ... the first inclined portion, the second inclined portion,
14a... first display unit, second display unit,
14b... first voice input unit, second voice input unit,
16... First imaging unit, second imaging unit,
43 ... motion determination means,
47 ... tilt control means,
S... Telepresence system.

Claims (3)

a first imaging unit that acquires an image of the first user; a first audio input unit that acquires the audio of the first user; a first display unit that displays the image; a first terminal having a first slope that slopes the display;
A second terminal that is installed at a location separated from the first terminal and that is capable of transmitting and receiving information to and from the first terminal, and that acquires an image of a second user. a shooting unit, a second audio input unit that acquires the voice of the second user, a second display unit that displays an image, and a second tilt unit that tilts the second display unit a second terminal having
Action discrimination means for discriminating respective actions of the first user and the second user based on images taken by the first imaging unit and the second imaging unit ;
One user's voice is input to one of the first terminal and the second terminal, and the other user performs a predetermined operation on the other of the first terminal and the second terminal is performed, the inclined portion of one of the first terminal and the second terminal is inclined according to the motion of the other user, and the first terminal and the second terminal tilt control means for not operating the tilt portion of the other terminal of the
A telepresence system comprising: said tilt control means for tilting one of said first terminal and said second terminal in a vertical direction when the other user of said first terminal and said second terminal performs a nodding motion; The telepresence system of claim 1, comprising: 3. The apparatus according to claim 1 or 2, wherein when the other terminal of the first terminal and the second terminal acquires a voice that does not reach a predetermined time, the voice input is ignored. A telepresence system as described.

Images